Understanding SQL Server Recovery Models

The #1 task that every DBA needs to be able to perform perfectly is to recover a database should it become corrupted for some reason. The recovery model of a database determines the options a DBA has when recovering a database. If a DBA can’t recover a database when a disaster strikes, then they better dust off their resume and start looking for a new line of work.

Part of a good database recovery plan is to first understand what recovery models are available. The recovery model of a database determines the types of database backups that can be taken, which in turn determines the different points in time in which a database can be restored.

Understanding the SQL Server Recovery Models

Each database within SQL Server has a recovery model setting. SQL Server has three different recovery models: Simple, Full, and Bulk-Logged. The recovery model setting determines what backup and restore options are available for a database, as well as how the database engine handles storing transaction log records in the transaction log.

The transaction log is a detailed log file that is used to record how the database is updated for each transaction. The SQL Server engine uses this log to maintain the integrity of the database. In the event that a transaction needs to be backed out, or the database crashes for some reason, the transaction log is used to restore the database to a consistent state. The recovery model for a database determines how much data the SQL Server engine needs to write to the transaction log, and whether or not a point-in-time restore can be performed. The initial recovery model setting for a new database is set based on the recovery model of the model system database. A database’s recovery model setting can be changed easily, by either using SSMS or TSQL code. To better understand the details of each one of these recovery models and how they affect the backup and restore options available, let me review each of the available recovery models.

Simple Recovery Model

The simple recovery model is the most basic of recovery models. When this recovery model is used, each transaction is still written to the transaction log. The transaction logs records will eventually be removed automatically when using the simple recovery model. That removal process happens for all completed transactions when a checkpoint occurs. Because log records are removed when a checkpoint occurs, transaction log backups are not supported when using the simple recovery model. This means point-in-time restores cannot be performed when a database has its recovery model set to SIMPLE. Because the transaction log is automatically cleaned up in this mode, this helps keep the transaction log small and from growing out of control.

With the simple recovery model, you can only perform full and differential backups. Because the simple recovery model doesn’t support using transaction log backups, you can only restore a database to the point-in-time when a full or differential backup has completed. If a database needs to support being restored to a point-in-time other than when a full or differential backup completes, then a different recovery model needs to be used.

Full Recovery Model

Just like it sounds, the full recovery model supports all the options for backing up and restoring a database. Just like the simple recovery model, all transactions are written to the transaction log, but unlike the simple recovery model, they stay in the transaction log after the transaction is completed. The transaction log records stay in the transaction log until a log backup is performed. When a transaction log backup is performed against a database that is in full recovery mode, the log records are written to the transaction log backup, and the completed transaction log records are removed from the transaction log.

Since every transaction is being written to the transaction log, the full recovery model supports point-in-time restores, meaning a database that is fully logged can be restored to any point in time. But it also means the transaction log needs to be big enough to support logging of all the transactions until a transaction log backup is run. If an application performs lots of transactions, there is the possibility that the transaction log will become full. When the transaction log becomes full, the database stops accepting transactions until a transaction log backup is taken, the transaction log is expanded, or the transaction log is truncated. Therefore, when a database uses the full recovery model, you need to ensure transaction log backups are taken frequently enough to remove the completed transactions from the transaction log before it fills up.

In addition to inserts and update transaction filling up the log, other operations like index create/alter and bulk load operations also write lots of information to the transaction log. If you find that your transaction log keeps filling up due to the index and bulk load operations (such as SELECT INTO), you might consider switching to the bulk-logged recovery model while these operations are being performed.

Bulk-Logged Recovery Model

The bulk-logged recovery model minimizes transaction log space usage when bulk-logged operations like BULK INSERT, SELECT INTO, or CREATE INDEX are executed. Bulk-logged recovery model functions similar to the full recovery model with the exception that transactions log records are minimally logged while bulk-logged operations are running. Minimal logging helps keep the log smaller, by not logging as much information.

Bulk-Logged recovery model improves the performance of large bulk loading operations by reducing the amount of logging performed. Additionally, because bulk-logged transactions are not fully logged, it reduces the amount of space written to the transaction log, which reduces the chance of the transaction log running out of space. Because bulk-logged operations are minimally logged, it affects point-in-time recoveries.

Point-in-time recoveries can still be performed when using the bulk-logged recovery model in some situations. If the database should become damaged while a minimal bulk-logged operation is being performed, the database can only be recovered to the last transaction log backup created prior to the first bulk-logged operation. When the transaction log backup contains a bulk logged operation the stopat options cannot be used. If no bulk-logged operations are performed at all while a database is using the bulk-logged recovery model, then you can still do a point-in-time restore just like you an in the full recovery model.

The bulk-logged recovery model is a great way to minimize transaction log space and improve the performance of large bulk update operations. But keep in mind, during the time a bulk-load operation has occurred, a point-in-time restore cannot be done. Therefore once the bulk-load operations are completed, a transaction log backup should be taken. By doing this, a point-in-time recovery can be made for transactions after this new transaction log backup is taken.

Which Recovery Model is being Used?

There are multiple ways to determine which recovery model is being used. One option is to use SQL Server Management Studio tool to find the recovery model of a database. To do this, first right-click on a database, then select the “Properties” item from the drop-down. Once the database properties are displayed, select the “Options” item from the left context menu. When this is done, the window in Figure 1 will display.

Figure 1: Recovery Model option

Figure 1 shows the Recovery Model setting of Simple for the AdventureWorks2017 database.

Another way to display the recovery options for a database is to run the TSQL code found in Listing 1.

Listing 1: Code to Display Recovery Model

SELECT name, recovery_model_desc  
FROM sys.databases  
WHERE name = 'AdventureWorks2017' ;

Changing the Recovery Model

Over time the recovery model for a database might need to change. This might occur when an application requires more or fewer recovery options for a given database. If the recovery model needs to be changed, it can easily be changed by running the code in Listing 2.

Listing 2: Changing the recovery model with TSQL code

USE master; 
GO
ALTER DATABASE AdventureWorks2017 SET RECOVERY FULL;
GO

In Listing 2, the database recovery model of the AdventureWorks2017 database was changed to FULL. Additionally, the Database Properties page of SSMS, as shown in Figure 1, can be used to change the database recovery model of a database. To change the recovery model, just select the correct option for the Recovery Model field and then click on the OK button.

Restoring Databases Using Each Recovery Model

The recovery model determines the kind of backups that can be taken for a database, which in turn determines what options you have for recovering a corrupted database. Let me explore how each recovery model might affect the restore options for a common problem that occurs, which causes a database to become corrupted. The common problem I will explore is when a TSQL programmer incorrectly updates a database and then asks the DBA to restore the database to some point-in-time prior to the database being corrupted by the erroneous update process. In the real world, a DBA will more than likely be required to perform more database restores due to someone running code that corrupts a database than having a database become corrupt due to a hardware failure. In the sections below, I will explore at least one option that could be used to recover a corrupted database due to a programming error for each of the different recovery models.

Simple Recovery Model

When a database is using simple recovery model, you can use full backup or a full and a differential backup to restore a database when it becomes corrupted, like in the event of a programming error. Since transaction log backups cannot be taken while using the simple recovery model, no point-in-time recoveries can be performed except for the end of a differential if available. Therefore, to recover a corrupted database, you need to restore the full database backup and possibly a differential database back that was taken just prior to the corruption occurring. Restoring full and differential backups leave the database in the state it was in when the backups were taken. All updates performed to the database since these backups completed will be missing from the database along with the updates performed by the programming error.

Full Recovery Model

Unlike the simple recovery model, the full recovery model offers multiple ways to recover from a bad update or deletion corrupting the database. The first option is to use the last full backup. If the full backup is used, then the recovered database would lose the same amount of updates as the simple recovery model restore in the prior section.

Another option is to use the point-in-time recovery options available with the full recovery model. Doing a point-in-time recovery means a DBA can recover the database to the point-in-time (to the minute) just before the erroneous update statements were performed. Any updates that were done to the database after the full backup but before the restore point will not be lost, unlike when using the simple recovery model.

In order to restore a database to a point in time after the full backup, a database needs at least one transaction log backup taken after the full backup. If a transaction log has not been taken prior to the corruption occurring, then the only option to remove the database corruption would be to restore the database using only the last full backup. In this case, the amount of data loss would be the same as the simple recovery model.

Keep in mind other options could be used to restore to a point-in-time after the full backup. One of those options would be to use a differential backup. A differential backup is a backup that contains all the changes to a database since the last full backup.

Bulk-Logged Recovery Model

Just like with the full recovery model, the bulk-logged recovery model does support point-in-time restores, as long as the point in time of the recovery is not contained within a transaction log backup that contains a bulk-logged operation. Therefore, if a database gets corrupted by a programming error while using the bulk-logged recovery model, a DBA could still recover the database as long as the corruption occurs before or after the transaction log backup that contains a bulk-logged operation. If the update does occur while a bulk-logged operation is being performed, then the best a DBA can do is to restore to the point-in-time of the last transaction log backup taken prior to the bulk-logged operation. Even though a DBA might not be able to restore to any point in time covered by a transaction log backup when the transaction log backup contains a bulk-logged operation, at least they are able to perform point-in-time recoveries as long as the transaction log doesn’t contain any bulk-logged operations.

Recovery Models

The recovery model of a database determines what backup/restore options are available. It also tells SQL Server how the transaction log should be managed. The recovery model also determines what recovery options are available for a database. With the simple recovery model, the only point that a database can be restored to is the last full backup or a differential. Whereas the full and bulk-logged recovery models allow a database to be recovered to a point after the full backup using a point-in-time restore. It is important to make sure each database on an instance has the appropriate recovery model setting based on the backup and restore requirements of the applications that use a database. Only by understanding the application backup and recovery requirements can the recovery model be set appropriately. If you want to understand more about backups and recovering a database using the different recovery models then, I’d suggest you consider reading about backup and restore options available in the Microsoft documentation that can be found here.

 

The post Understanding SQL Server Recovery Models appeared first on Simple Talk.

from Simple Talk https://ift.tt/3jk5rHg
via

Build a REST API in .NET Core

One way to scale large complex solutions is to break them out into REST microservices. Microservices unlock testability and reusability of business logic that sits behind an API boundary. This allows organizations to share software modules because REST APIs can be reused by multiple clients. Clients can then call as many APIs from mobile, web, or even static assets via a single-page app.

In this take, I will show you what it takes to build a REST API in .NET Core. I will hit this with real-world demands such as versioning, search, and logging, to name a few. REST is often employed with verbs like POST, PUT, or PATCH, so I plan to cover them all. What I hope you see is a nice, effective way to deliver value with the tools available.

This article assumes a working grasp of ASP.NET, C#, and REST APIs so I will not cover any basics. I recommend the latest .NET Core LTS release at the time of this writing to follow along. If you would like to start with working code, the sample code can be downloaded from GitHub.

You can begin by creating a new folder like BuildRestApiNetCore and firing this up in a shell:

dotnet new sln
dotnet new webapi --no-https
dotnet sln add .

This project is based on a Web API template with HTTPS disabled to make it easier for local development. Double-clicking the solution file brings it up in Visual Studio if you have it installed. For .NET Core 3.1 support, be sure to have the 2019 version of the IDE.

APIs put a layer of separation between clients and the database, so the data is an excellent place to start. To keep data access trivial, Entity Framework has an in-memory alternative so that I can focus on the API itself.

An in-memory database provider comes via NuGet:

dotnet add package Microsoft.EntityFrameworkCore.InMemory

Then, create the following data model. I put this in the Models folder to indicate this namespace houses raw data. To use the data annotations, add System.ComponentModel.DataAnnotations in a using statement.

public class Product
{
  [Key]
  [Required]
  [Display(Name = "productNumber")]
  public string ProductNumber { get; set; }

  [Required]
  [Display(Name = "name")]
  public string Name { get; set; }

  [Required]
  [Range(10, 90)]
  [Display(Name = "price")]
  public double? Price { get; set; }

  [Required]
  [Display(Name = "department")]
  public string Department { get; set; }
}

In a real solution, this may go in a separate project depending on the team’s needs. Pay attention to the attributes assigned to this model like Required, Display, and Range. These are data annotations in ASP.NET to validate the Product during model binding. Because I use an in-memory database, Entity Framework requires a unique Key. These attributes assign validation rules like price range or whether the property is required.

From a business perspective, this is an e-commerce site with a product number, name, and price. Each product is also assigned a department to make searches by department easier.

Next, set the Entity Framework DbContext in the Models namespace:

public class ProductContext : DbContext
{
  public ProductContext(DbContextOptions<ProductContext> options) : base(options)
  {
  }

  public DbSet<Product> Products { get; set; }
}

This database context is the dependency injected in the controller to query or update data. To enable Dependency Injection in ASP.NET Core, crack open the Startup class and add this in ConfigureServices:

services
  .AddDbContext<ProductContext>(opt => 
              opt.UseInMemoryDatabase("Products"));

This code completes the in-memory database. Be sure to add Microsoft.EntityFrameworkCore to both classes in a using statement. Because a blank back end is no fun, this needs seed data.

Create this extension method to help iterate through seed items. This can go in an Extensions namespace or folder:

public  static class EnumerableExtensions
{
  public static IEnumerable<T> Times<T>(this int count, Func<int, T> func)
  {
    for (var i = 1; i <= count; i++) yield return func.Invoke(i);
  }
}

The initial seed goes in Models via a static class:

public static class ProductSeed
{
  public static void InitData(ProductContext context)
  {
    var rnd = new Random();

    var adjectives = new [] { "Small", "Ergonomic", "Rustic", 
                                        "Smart", "Sleek" };
    var materials = new [] { "Steel", "Wooden", "Concrete", "Plastic",
                                       "Granite", "Rubber" };
    var names = new [] { "Chair", "Car", "Computer", "Pants", "Shoes" };
    var departments = new [] { "Books", "Movies", "Music", 
                                       "Games", "Electronics" };

    context.Products.AddRange(900.Times(x =>
    {
      var adjective = adjectives[rnd.Next(0, 5)];
      var material = materials[rnd.Next(0, 5)];
      var name = names[rnd.Next(0, 5)];
      var department = departments[rnd.Next(0, 5)];
      var productId = $"{x, -3:000}";

      return new Product
      {
        ProductNumber = 
           $"{department.First()}{name.First()}{productId}",
        Name = $"{adjective} {material} {name}",
        Price = (double) rnd.Next(1000, 9000) / 100,
        Department = department
      };
    }));

    context.SaveChanges();
  }
}

This code loops through a list of 900 items to create this many products. The names are picked at random with a department and price. Each product gets a “smart” key as the primary key which comes from department, name, and product id.

Once this seed runs, you may get products such as “Smart Wooden Pants” in the Electronics department for a nominal price.

As a preliminary step to start building endpoints, it is a good idea to set up versioning. This allows client apps to upgrade API functionality at their leisure without tight coupling.

API versioning comes in a NuGet package:

dotnet add package Microsoft.AspNetCore.Mvc.Versioning

Go back to the Startup class and add this to ConfigureServices:

services.AddApiVersioning(opt => opt.ReportApiVersions = true);

I opted to include available versions in the API response, so clients know when upgrades are available. I recommend using Semantic Versioning to communicate breaking changes in the API. Letting clients know what to expect between upgrades helps everyone stay on the latest features.

Search endpoint in a REST API

To build an endpoint, spin up a Controller in ASP.NET which goes in the Controllers folder.

Create a ProductsController with the following, making sure to add the namespace Microsoft.AspNetCore.Mvc with a using statement:

[ApiController]
[ApiVersion("1.0")]
[Route("v{version:apiVersion}/[controller]")]
[Produces("application/json")]
public class ProductsController : ControllerBase
{
  private readonly ProductContext _context;

  public ProductsController(ProductContext context)
  {
    _context = context;

    if (_context.Products.Any()) return;

    ProductSeed.InitData(context);
  }
}

Note InitData runs the initial seed when there aren’t any products in the database. I set a Route that uses versioning which is set via ApiVersion. The data context ProductContext gets injected in the constructor with Dependency Injection. The first endpoint is GET which returns a list of Products in the Controller:

[HttpGet]
[Route("")]
[ProducesResponseType(StatusCodes.Status200OK)]
public ActionResult<IQueryable<Product>> GetProducts()
{
  var result = _context.Products as IQueryable<Product>;

  return Ok(result
    .OrderBy(p => p.ProductNumber));
}

Be sure to add Microsoft.AspNetCore.Http in a using statement to set status codes in the response type.

I opted to order products by product number to make it easier to show the results. In a production system check, this sort matches the clustered index, so the database doesn’t have to work as hard. Always review execution plans and statistics IO to confirm good performance.

This project is ready to go for a test drive! Inside of a CLI type:

dotnet watch run

Hit the endpoint with curl:

curl -i -X GET "http://localhost:5000/v1/products" -H "accept: application/json"

I run both commands in separate consoles. One runs the file watcher that automatically refreshes when I make changes. The other terminal is where I keep curl results. Postman is also useful, but curl gets the job done and comes with Windows 10.

Below are the results:

T his request returns all products in the database, but it’s not scalable. As the product list grows, clients will get slammed with unbound data, p utting more pressure on SQL and network traffic.

A better approach is to introduce limit and offset request parameters in a model:

public class ProductRequest
{
  [FromQuery(Name = "limit")]
  public int Limit { get; set; } = 15;

  [FromQuery(Name = "offset")]
  public int Offset { get; set; }
}
Wire this request parameter to the GetProducts endpoint:
public ActionResult<IQueryable<Product>> 
             GetProducts([FromQuery] ProductRequest request)
{
  // ...

  Response.Headers["x-total-count"] = result.Count().ToString();

  return Ok(result
    .OrderBy(p => p.ProductNumber)
    .Skip(request.Offset)
    .Take(request.Limit));
}

Note I set an HTTP header x-total-count with the Count. This helps clients that may want to page through the entire result set. When requests parameters are not specified then the API defaults to the first 15 items.

Next, add a search parameter to filter products by department:

public ActionResult<IQueryable<Product>> GetProducts([FromQuery] 
             string department, [FromQuery] ProductRequest request)
{
  // ...

  if (!string.IsNullOrEmpty(department))
  {
    result = result.Where(p => p.Department.StartsWith(department, 
                 StringComparison.InvariantCultureIgnoreCase));
  }

  // ..
}

Search can go inside a conditional block that alters the query. Note I use StartsWith and InvariantCultureIgnoreCase to make it easier to filter products. In actual SQL, the LIKE operator is useful, and case insensitivity can be set via collation.

To test out paging and this new filter in curl:

curl -i -X GET http://localhost:5000/v1/products?offset=15&department=electronics
  -H "accept: application/json"

Be sure to check out HTTP headers which include total count and supported versions:

HTTP/1.1 200 OK
Date: Sat, 11 Jul 2020 18:35:07 GMT
Content-Type: application/json; charset=utf-8
Server: Kestrel
Content-Length: 1459
x-total-count: 194
api-supported-versions: 1.0

Logging and API Documentation

With the API taking shape, how can I communicate endpoints to other developers? It is beneficial for teams to know what the API exposes without having to bust open code. Swagger is the tool of choice here; by using reflection, it is capable of documenting what’s available.

What if I told you everything swagger needs is already set in this API? Go ahead, take a second look:

[Produces("application/json")]
[ProducesResponseType(StatusCodes.Status200OK)]
ActionResult<IQueryable<Product>> GetProducts([FromQuery] 
               string department, [FromQuery] ProductRequest request)

ASP.NET attributes are useful for documenting endpoints. Swagger also picks up return types from controller methods to figure out what responses look like and picks up request parameters in each controller method via reflection. It produces “living documentation” because it sucks up everything from working code, which reduces mishaps.

The one dependency lacking is a NuGet:

dotnet add package Swashbuckle.AspNetCore

And wire this up in ConfigureServices:

services.AddSwaggerGen(c => c.SwaggerDoc("v1", new OpenApiInfo
{
  Title = "Products",
  Description = "The ultimate e-commerce store for all your needs",
  Version = "v1"
}));

Then, enable this in Configure:

app.UseSwagger();
app.UseSwaggerUI(opt => opt.SwaggerEndpoint("/swagger/v1/swagger.json", "Products v1"));

Note OpenApiInfo comes from the Microsoft.OpenApi.Models namespace. With this, navigate to http://localhost:5000/swagger in the browser to check out the swagger doc.

The page should like this:

From the swagger doc, feel free to poke around and fire requests to the API from this tool. Fellow developers from across the organization might even buy you a cup of coffee for making their lives easier.

Note how expanding GET /Products picks up C# data types from the method in the controller:

The next stop is the logger. I will use NLog to store logs in the back end. This enables the API to save logs for further analysis. In a real environment, logs are useful for troubleshooting outages. They also aid in gathering telemetry to help understand how the API is utilized in the wild.

To set up the logger, I am going to need the following:

• A NuGet package
• An nlog.config settings file
• Changes in the Program class
• Tweaks in appsettings.json

To install the NuGet package:

dotnet add package NLog.Web.AspNetCore

The nlog.config file can be:

<?xml version="1.0" encoding="utf-8" ?>
<nlog xmlns=”http://www.nlog-project.org/schemas/NLog.xsd”
      xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance”
      throwExceptions="false"
      throwConfigExceptions="false"
      autoReload="true"
      internalLogLevel="Warn"
      internalLogFile=
           "C:\temp\BuildRestApiNetCore\RestApi-internal-nlog.txt">

  <extensions>
    <add assembly="NLog.Web.AspNetCore"/>
  </extensions>

  <targets async="true">
    <target xsi:type="File"
            name="ownFile-web"
            fileName=
              "C:\temp\BuildRestApiNetCore\RestApi-${shortdate}.log">

      <layout xsi:type="JsonLayout">
        <attribute name="Timestamp" layout="${longdate}" />
        <attribute name="Level" layout="${uppercase:${level}}" />
        <attribute name="Logger" layout="${logger}" />
        <attribute name="Action" layout="${aspnet-mvc-action}" />
        <attribute name="Message" layout="${message}" />
        <attribute 
           name="Exception" layout="${exception:format=tostring}" />
      </layout>
    </target>
  </targets>

  <rules>
    <logger name="Microsoft.*" maxlevel="Info" final="true" /> 
                <!-- blackhole -->
    <logger name="*" minlevel="Info" writeTo="ownFile-web" />
  </rules>
</nlog>

Pay attention to Layout because it sets the type of log file which is set to JsonLayout. This JSON format has the most flexibility when consuming log files in different analytical tools. Logger rules do not log errors from Microsoft.* to keep chattiness down to a minimum. As a bonus, unhandled exceptions from the API get logged but do not rethrow because throwExceptions is false. Usage here may vary, but it is generally a good idea to handle all unhandled exceptions in the logger.

Inside the Program class, enable NLog, remembering to include using NLog.Web:

Host.CreateDefaultBuilder(args)
  .ConfigureWebHostDefaults(webBuilder =>
  {
    webBuilder.UseStartup<Startup>();
  })
  .UseNLog();

Finally, make these tweaks to configure logging in appsettings.json:

"Logging": {
  "LogLevel": {
    "Default": "Information",
    "Microsoft": "None",
    "Microsoft.AspNetCore": "Error",
    "Microsoft.Hosting.Lifetime": "Information"
  }
}

The basic idea is to cut the number of log entries which aren’t relevant to this API. Feel free to poke around with the settings, so it logs exactly what the API needs.

It’s time to take this for a spin. I n the Controller class, add using Microsoft.Extensions.Logging and inject a plain old ASP.NET logger:

private readonly ILogger<ProductsController> _logger;

public ProductsController(ProductContext context, 
            ILogger<ProductsController> logger)
{
  _logger = logger;
  // ...
}

Say now the team decides to grab telemetry around how often clients ask for 100 records or more.

Put this inside GetProducts:

if (request.Limit >= 100)
  _logger.LogInformation("Requesting more than 100 products.");

Be sure to have a temp folder handy to check the logs, for example, C:\temp\BuildRestApiNetCore\.

This is what an entry might look like:

{
  "Timestamp": "2020-07-12 10:30:30.8960",
  "Level": "INFO",
  "Logger": "BuildRestApiNetCore.Controllers.ProductsController",
  "Action": "GetProducts",
  "Message": "Requesting more than 100 products."
}

REST Endpoints with Verbs

Take a deep breath in and breath out. This API is almost production-ready with minimal code. I will now quickly turn towards REST features such as POST, PUT, PATCH, and DELETE.

The POST endpoint takes in a body with the new product and adds it to the list. This method is not idempotent because it creates new resources when invoked.

Put this in ProductsController:

[HttpPost]
[ProducesResponseType(StatusCodes.Status201Created)]
[ProducesResponseType(StatusCodes.Status400BadRequest)]
public ActionResult<Product> PostProduct([FromBody] Product product)
{
  try
  {
    _context.Products.Add(product);
    _context.SaveChanges();

    return new CreatedResult($"/products/{product.ProductNumber.ToLower()}", product);
  }
  catch (Exception e)
  {
    _logger.LogWarning(e, "Unable to POST product.");

    return ValidationProblem(e.Message);
  }
}

ASP.NET automatically handles exceptions via ValidationProblem. This validation returns an RFC 7807 spec compliant response with a message. In a real system, I recommend making sure this does not expose any internals about the API. Putting the exception message here helps clients troubleshoot their code, but security is also important. I opted to include the error message mostly for demonstration purposes. The exception is also logged as a warning, to avoid logging a bunch of errors. Monitoring tools might page out to whoever is on-call when there are too many exceptions. A best practice is to only log errors during catastrophic failures that might need human intervention.

Using the swagger tool, the curl command is:

curl -i -X POST http://localhost:5000/v1/products
  -H "accept: application/json"
  -H "Content-Type: application/json"
  -d "{\"productNumber\":\"string\",\"name\":\"string\",\"price\":10,\"department\":\"string\"}"

When there are problems with the request, the API responds with:

{
  "errors": {},
  "type": "https://tools.ietf.org/html/rfc7231#section-6.5.1",
  "title":"One or more validation errors occurred.",
  "status": 400,
  "detail": "An item with the same key has already been added. Key: string",
  "traceId":"|c445a403-43564e0626f9af50."
}

A 400 (Bad Request) response indicates a user error in the request. Because users can’t be trusted to send valid data, the API logs a warning.

Note that on success POST returns a 201 with Location:

HTTP/1.1 201 Created
Date: Mon, 13 Jul 2020 22:52:46 GMT
Content-Type: application/json; charset=utf-8
Server: Kestrel
Content-Length: 76
Location: /products/bc916
api-supported-versions: 1.0

This points the client towards the new resource. So, it is a good idea to spin up this GET endpoint:

[HttpGet]
[Route("{productNumber}")]
[ProducesResponseType(StatusCodes.Status200OK)]
[ProducesResponseType(StatusCodes.Status404NotFound)]
public ActionResult<Product> GetProductByProductNumber([FromRoute] 
            string productNumber)
{
  var productDb = _context.Products
    .FirstOrDefault(p => p.ProductNumber.Equals(productNumber, 
              StringComparison.InvariantCultureIgnoreCase));

  if (productDb == null) return NotFound();

  return Ok(productDb);
}

A 404 response indicates the resource does not exist in the API yet but might become available at some point in the future.

PUT is similar:

[HttpPut]
[ProducesResponseType(StatusCodes.Status200OK)]
[ProducesResponseType(StatusCodes.Status404NotFound)]
[ProducesResponseType(StatusCodes.Status400BadRequest)]
public ActionResult<Product> PutProduct([FromBody] Product product)
{
  try
  {
    var productDb = _context.Products
      .FirstOrDefault(p => p.ProductNumber.Equals(product.ProductNumber, 
           StringComparison.InvariantCultureIgnoreCase));

    if (productDb == null) return NotFound();

    productDb.Name = product.Name;
    productDb.Price = product.Price;
    productDb.Department = product.Department;
    _context.SaveChanges();

    return Ok(product);
  }
  catch (Exception e)
  {
    _logger.LogWarning(e, "Unable to PUT product.");

    return ValidationProblem(e.Message);
  }
}

In REST design, a PUT allows updates to an entire resource. It is idempotent because multiple identical requests do not alter the number of resources.

Like a GET 404 response, the resource is unavailable for updates, but this might change later. As a bonus, ASP.NET provides model binding validation out of the box. Go ahead, try to update an existing resource with bad data.

This JSON is the Bad Request response you might see:

{
  "errors": {
    "Price": ["The price field is required."]
  },
  "type": "https://tools.ietf.org/html/rfc7231#section-6.5.1",
  "title": "One or more validation errors occurred.",
  "status": 400,
  "traceId": "|c445a409-43564e0626f9af50."
}

PATCH is the most complex of all verbs because it only updates a part of the resource via a JSON Patch document.

The good news is .NET Core helps with a NuGet package:

dotnet add package Microsoft.AspNetCore.Mvc.NewtonsoftJson

Then, enable this in ConfigureServices:

services.AddControllers().AddNewtonsoftJson();

This is the PATCH endpoint. Remember using Microsoft.AspNetCore.JsonPatch:

[HttpPatch]
[Route("{productNumber}")]
[ProducesResponseType(StatusCodes.Status200OK)]
[ProducesResponseType(StatusCodes.Status404NotFound)]
[ProducesResponseType(StatusCodes.Status400BadRequest)]
public ActionResult<Product> PatchProduct([FromRoute] 
      string productNumber, [FromBody] JsonPatchDocument<Product> patch)
{
  try
  {
    var productDb = _context.Products
      .FirstOrDefault(p => p.ProductNumber.Equals(productNumber, 
           StringComparison.InvariantCultureIgnoreCase));

    if (productDb == null) return NotFound();

    patch.ApplyTo(productDb, ModelState);

    if (!ModelState.IsValid || !TryValidateModel(productDb)) 
             return ValidationProblem(ModelState);

    _context.SaveChanges();

    return Ok(productDb);
  }
  catch (Exception e)
  {
    _logger.LogWarning(e, "Unable to PATCH product.");

    return ValidationProblem(e.Message);
  }
}

I hope you see a pattern start to emerge with the different status code response types. A 200 OK means success and a 400 Bad Request means user error. Once a patch gets applied it appends any validation errors in ModelState. Take a closer look at JsonPatchDocument, which does model binding, and ApplyTo, which applies changes. This is how a JSON Patch document gets applied to an existing product in the database. Exceptions get logged and included in the response like all the other endpoints. A 404 (Not Found) response indicates the same situation as all the other verbs. This consistency in response status codes helps clients deal with all possible scenarios.

A JSON patch request body looks like the following:

[{
  "op": "replace",
  "path": "price",
  "value": 13.67
}]

Model binding validation rules still apply to the patch operation to preserve data integrity. Note the patch gets wrapped around an array, so it supports an arbitrary list of operations.

This is PATCH in curl:

Last stop, a DELETE method:

[HttpDelete]
[Route("{productNumber}")]
[ProducesResponseType(StatusCodes.Status200OK)]
[ProducesResponseType(StatusCodes.Status404NotFound)]
public ActionResult<Product> DeleteProduct([FromRoute] 
        string productNumber)
{
  var productDb = _context.Products
    .FirstOrDefault(p => p.ProductNumber.Equals(productNumber, 
           StringComparison.InvariantCultureIgnoreCase));

  if (productDb == null) return NotFound();

  _context.Products.Remove(productDb);
  _context.SaveChanges();

  return NoContent();
}

The status code response is No Content:

HTTP/1.1 204 No Content
Date: Tue, 14 Jul 2020 22:59:20 GMT
Server: Kestrel
api-supported-versions: 1.0

This status signals to clients that the resource is no longer available because the response body is empty. The response can also be 204 (Accepted) if this needs an async background process to clean up the data. In a real system, soft deletes are sometimes preferable to allow rollback during auditing. When deleting data, be sure to comply with GPDR or any policy that applies to the data.

Conclusion

.NET Core adds many useful features to your toolbelt to make working with REST APIs easier. Complex use cases such as documentation, validation, logging, and PATCH requests are simpler to think about.

The post Build a REST API in .NET Core appeared first on Simple Talk.

from Simple Talk https://ift.tt/3jcH1zg
via

DBA in training: Preparing for interviews

The series so far:

1. DBA in training: So, you want to be a DBA...
2. DBA in training: Preparing for interviews

Ah, the big black box – the technical interview. You’ve heard about them, but maybe you have never been through one – and it feels like a final exam without a study guide. Let’s take a moment to discuss the different interview scenarios, the best way to prepare, and some ways of answering the “What is your biggest weakness” and the “What questions do you have” scenarios.

Preparation begins long before you submit applications. Start by downloading SQL Server Developer Edition, installing the AdventureWorks database, or the StackOverflow database – or make your own. Then start working on your TSQL. There are a wealth of resources on the internet to help – both free and low cost. Look at RedGate, SQLSkills, Brent Ozar, Kathi Kellenberger, Bert Wagner (to name a few!). If you are confident in your TSQL skills, then work on actual database management. Ensure that you understand the rudiments of backups, restores, RPO and RTO, security and internals. If you can confidently write queries, manage automated backups and restores, properly write and maintain indexing and troubleshoot issues on your practice instance, that is one step toward being able to manage many instances. If any of this is Greek to you, don’t worry – I will cover these later.

Overall preparation

Hopefully, you have read up on the company when you submitted your application. You should have a basic understanding of what they do and what they are trying to achieve. That can help you to better understand how you can contribute toward that goal.

But what if I have never worked as a DBA in my life! How can I help them?

I am glad you asked! Yes, you’re green, but that doesn’t mean you have nothing to offer. Consider how you have contributed toward the growth of other places you have worked. Can you show where you went above and beyond to meet the need of past positions you held? If you went to school or boot camp, do you have internship experience you can cite? Maybe you have worked in a related field and have experience with TSQL or SAN management that you can leverage.

You may have gone to college straight out of school. In that case, you may have to think a little more creatively. Did you do any volunteer work or projects with databases? How can you demonstrably show your drive and passion for excelling in this field?

When I was breaking into this profession, I did have a work history – just nothing in the technical field. I had to rely on great personal recommendations and what I could come up with to show my passion and ability to be taught. My capstone project in college revolved around SQL Server (and Access). I ensured that I had mentors to guide me. I bought books and began reading whatever blogs I could get my hands on to prepare myself. I won’t lie; I had to submit a lot of applications before the right opportunity came along. If it had not, I would have bloomed where I was planted and waited for my chance while I continued to teach myself. Whenever I see someone going out of their way from the love for what they want to do, it gets my attention, and I don’t believe I am alone in that observation.

Finally, do you have background experience in the company’s field? For instance, if you have applied to be a DBA at a hospital, do you have clinical experience? If you are applying to work in the financial industry, do you have an understanding of business practices? Especially at small companies, having a DBA who understands not only SQL Server internals, but who can also speak to the business users in language that they can understand may provide an advantage that is of interest to the company.

Before I was hired as a DBA, I submitted an application to write reports for the place where I was presently working. After I got the DBA job, my current employer found out (too late) about my application. They would have hired me for the new position, but my application got lost in the shuffle and wasn’t discovered in time. Why would they have hired me for a technical job that I had no experience doing? I had five years’ worth of knowledge of the company, where I had worked for five years. The business users would have wanted that bridge between the business and the technical. Every little bit helps.

I was once helping to interview candidates for an ETL position. One of my favorite questions was, “Tell me about the most challenging project you ever took ownership of, and how you saw it through.” One of the best answers I ever received was from a candidate who told me that she was assigned a project that included technology she had never seen before. Much of it was proprietary, and there was no documentation to reference. The person who had been previously working on it had left the company some time before. Seeing it through required working long hours of overtime for months. She tracked down the former employee, who agreed to meet with her and answer her questions in order to gain the knowledge she needed.

Needless to say, she stuck it out, completed the project and did it well. She was the one I recommended for the job, and she has excelled at it. Yes, she had technical experience, but it was her drive that made her a standout. Technical knowledge can be taught. Passion and drive cannot.

Do many companies look for (and need!) experienced DBAs? Yes, of course. Finding the company who is willing to teach a new DBA can really be the lottery ticket. You could easily find that you need to start in a related field, forge a relationship with a DBA on site who is willing to teach you, and wait for your big chance. You may have to start at the low end of the salary scale to get your foot in the door and earn your experience. If you are persistent – and lucky – you will arrive at the time where you perk someone’s interest enough to have a chat about giving you your big break. And that is where the interview comes in.

Interview Prep 101

Here is where having a mentor or a friend in the field can be invaluable. At the very least, you want a friend who can fire off some questions (if your friend is non-technical, have the answers ready for them!) and who will be brutally honest with you. Be likeable and relatable – and above all, please don’t be afraid to say, “I don’t know”, especially when you follow it up with an attempt at a solution. An example of this would be, “I don’t have any experience with managing AlwaysOn, so if I were asked to troubleshoot it, I would probably look at Alan Hirt’s blog for guidance, since he is an authority on the subject.” Or if you are not sure whom to look to for answers, you might say: “I am working to better learn about security. I would need to reference Stack Overflow and see if someone else has had that same issue.” No one can possibly know everything about SQL Server, and an interviewer should understand that. However, if they can see the wheels turning toward a resolution, and especially if they can see that you where to look, it will be a plus. Try to fake an answer, and you will be caught. It is frequently a deal-breaker, so don’t do that!

Types of interviews to expect

Technical interviews are usually a multistep process. Understanding the steps involved and their purpose will help you to be better prepared and more confident as you progress through the steps. Here is a brief rundown of what you might encounter.

Phone interviews

A phone interview is the basic getting-to-know-you assessment. You may be asked some technical questions, but many times, the purpose of the phone interview is to get an overall picture of where you are professionally, to be questioned about the items on your resume, and to determine how well you would fit in with the team. There may be more than one phone interview, or it could be one conference call where several people address you through the course of the conversation.

The on site interview

An onsite interview lasts anywhere from several hours to several days. You’ll run the gauntlet of peer interviews, meetings with managerial staff, and time with HR. Onsite interviews may take the form of verbal meetings, practical interviews where you may be asked to either diagram pseudocode or to work on an isolated instance of SQL Server to resolve problems you are given, or you may be required to complete online technical assessments. The company will want to confirm the degree of your technical expertise to their satisfaction and determine whether you will be a good fit for their needs. Don’t be surprised if the questions get really difficult – they may be checking the limits of your knowledge. You want to be the calm in the storm, and you want them to love you. However, you also need to like what you see. You will be investing a period of your life with this company – time you won’t get back. Try to get a feel for the culture and the people who are speaking with you.

As to the “What’s your biggest weakness” question. Don’t give a non-answer here. For instance, “Well, I find that I have a difficult time working less than ten hours a day”, or “I’m just too motivated” and answers of that ilk will be likely to earn you an eye roll once the interviewer exists the room. Be honest, but include what you are doing to remedy it. For example: “I have found that troubleshooting error messages has frequently led me down the wrong path, so I have started reading what Paul Randal has written about knee-jerk troubleshooting and am focusing on troubleshooting with wait statistics.” Or: “Isolation levels have been difficult for me to wrap my head around. Kendra Little wrote a blog on them that I am reading right now to better understand them.” If you are new? “I am recently out of school, so I haven’t yet worked in a professional environment. We didn’t cover SQL Server internals in any depth, so I am trying to become proficient at understanding as much of it as possible, as quickly as I can.” Be prepared to answer when they follow up by asking what you are doing to become proficient!

It is important to understand that the entire time you are onsite, whether you are in a formal interview situation or not, you are, in fact, still in an interview. Before I became a DBA, I worked as an assistant to a division director at my company. One small part of my duties involved welcoming the people he interviewed and chatting with them while they were waiting to speak to him, as well as escorting them to other interviews. After the formal interviews finished for the day, he invariably asked me for my impression of the candidates, and those impressions were taken seriously. Given the way some of the candidates spoke to me, it seemed clear that they were blissfully unaware of this reality.

You will most probably be asked what questions you have. This is not the time to say “I don’t know”. Even if you have never worked as a DBA a day in your life, you have questions. Many questions. Some possibilities?

• How many people are on the DBA team? If you are completely inexperienced, being the lone DBA on site is likely to be a baptism by fire – hardly optimal. Not to mention that you will be on call 24/7/365. If you are more experienced, you may want to ensure that the opportunity is commensurate with the increased burden of flying solo.
• How many SQL Server instances are there on site, and what version/edition are they running? You don’t need names of instances here (and shouldn’t get them). Here you want to know how old the versions are and whether they are running Express, Standard or Enterprise. If they are running versions of SQL Server that are no longer supported by Microsoft, your job could get stressful very quickly if something goes wrong. On the other hand, if all your experience is with Developer (which is the free version of Enterprise), you will have to rethink how you do some things with Express or Standard.
• Does the company have server monitoring tools? You may or may not be told what they are using, but it would be nice to know if they have anything, and if so, what you would be expected to work with.
• Does the company have support from Microsoft or any consultants? If not, would they consider that option? I have run into more than one case where consultants saved us a great deal of time and money helping us to solve esoteric problems that none of us had ever seen before. Some of the best consultants have access to information that you won’t have and are worth their weight in gold.
• Does the company have change management? You are hoping that they do – otherwise, deployments may look like the Wild West, and your job might be more interesting than you bargained for.
• How often does the company practice disaster recovery? Practice makes perfect, and you don’t want the first time to be when it’s real. For that matter, do they have a run book that you would have access to once you started? If they don’t, you will look good for having thought of it!
• What are three things that the company would expect of you in the first year to consider your employment a success? (Alternatively, ask: Please define a successful DBA at your organization). Take note of that answer and add it to your goals list.

The second interview

Congratulations, you’re a contender!

The second interview is frequently your final audition. This is when the company takes one last look prior to making the decision. You have met some of the staff and had a chance to get some feel for the company, and whether you might like the people you would be working with. You’ll probably get a run-down of benefits, a few questions, and an offer to ask any other questions you have. Here’s an important one to consider asking:

• It’s important for me to continue to learn and develop my professional skills in order to be able to offer you my best work. What avenues does the company offer for continuing education? (The open-ended phrasing is deliberate here. You want to hear what they are already doing. Above all, you don’t want promises that may be made with the best of intentions, only to fall through at the next budget meeting. Primarily, you are looking for conference attendance at events such as PASS Summit, DevIntersection or SQLSkills classes. If the company’s budget doesn’t allow for travel, you want to see that it encourages access to online classes, O’Reilly, or to curriculum such as that offered by sites like Pluralsight. If they ask what you had in mind, mention the alternatives here and see what kind of reaction you get. You have worked too hard to get this far, only to let your skills stagnate. By phrasing it this way, the company should understand that the process of lifelong learning brings value to them as well. A company that invests in its IT department is, generally speaking, a good place to work.
• Try to determine if the position is “compartmentalized”. How to find this out? Ask what a typical day on the job will look like for you. Larger companies can often have DBAs who just do one or two things. If that form of specialization appeals to you, it could be a good fit. If you need more variety, you need to see if that is available, or perhaps you would be better off looking at a smaller company.
• Is working from home something important to you? Now is a good time to ask.

Once you’ve finished the interview process, take a look at Brent Ozar’s annual salary survey, GlassDoor, and any other resources you can think of that would give salary information for a DBA in your geographic area and of your education and experience. If you are new, be prepared to go in at or toward the bottom of the scale and work your way up. The more competitive salary will come with experience and proof of your value. If you have the experience, this will help you to know if you are being offered a fair salary.

When you consider the offer, think about how your salary and professional growth will look at the company over the coming years. If you want to be a DBA all the days of your life, how far can you grow here? For instance, if a variety of experience is important to you but you think that you will at least be initially compartmentalized in your new position, will you be allowed a chance to do other things down the road? Is there room for promotion? If you are interested in management (which is an entirely different skillset), is there an opportunity for you to do that? With the cost of living increases, will your salary offer allow for a long-term relationship with the company? Try to gain as clear of a picture as possible of what your time with your prospective new company will look like before making your decision. You will be spending most of your waking weekday hours there, after all.

Contract work vs. FTE positions

Some companies prefer to either engage contract workers exclusively or as a “try before you buy” avenue to hiring full-time employees. Contract work frequently pays better, since they don’t have to deal with benefits, although when I did it, insurance was offered. It can be a good way to rack up experience if it is your first foray into the IT field or if you want part-time or full-time work. If it is a contract-to-hire position, it affords you the chance to try before you buy, too. The dream for most is a full-time job. Those jobs can be more difficult to find, but if you have the chance to work at several places, and if you cultivate a good reputation, you may find it helps to land that permanent position. The DBA world is a small one, and word spreads, especially if you are able to be active in your local SQL Server user group, and you may find that as you give, you receive as well.

Conclusion

This is a lot of information to digest, I know. Going into your interview well prepared will do much to show you at your best. It is the moment when preparation meets opportunity. I’ll sprinkle in my best wishes for your success and cross my fingers that the magic happens for you.

 

The post DBA in training: Preparing for interviews appeared first on Simple Talk.

from Simple Talk https://ift.tt/2FYzqpK
via

How to survive and thrive when watching online conferences

Think of online conferences in the same way you would an in-person event … and then think again.

Here at Redgate, there’s a strong emphasis on learning and development that continues in this new world where COVID-19 has stripped away the chance for us to meet face-to-face safely. As Redgate’s Events Marketing Manager, it’s been even more essential for me to dedicate time over the past few months building my knowledge of how to create the best possible event experience.

I’ve been reflecting on what I’ve learned when tuning into a variety of virtual online conferences and want to share some food for thought that I hope you’ll keep in mind for your next online event.

Consider the ROI

While an online conference will likely be a lot cheaper than it would be if you were attending in person, there’s still a key hidden cost – time. I’m a big believer in the saying that you get out what you put in. Registering for the event is only the tip of the iceberg, and if you want to get valuable insights from the speakers and content delivered, you’re going to need to work for it. Don’t expect that just because you’re not flying to the event, you can show up in your PJs. Dress for business, roll up your sleeves and get to work. This is work time, not downtime.

Prepare in advance

Know what you want to get out of the experience – what are your goals, for example? What do you want to learn? What problems are you hoping to solve? Write down the questions you want to be answered to help you discover how best to spend your time while tuning in.

Also, think about the particular sessions you really want to listen to live. Do you have a question that you want answered, maybe from a certain speaker or company? Is the platform offering the chance for you to network with peers or vendors? Should you allocate time for that and, if so, which session could you skip or catch up on later?

Prepare a calendar reminder for the event, outlining the sessions on the agenda you want to see. Or get yourself organized and set up a OneNote in advance with tabs, a checklist and space to add in notes from sessions.

Get set up

Ahead of the date, make sure you know how to log on and join the online event by checking your confirmation email for the right link to get access. Look at the tech that’s required and ensure you have what you need to hear or see things when they go live (decent internet is essential). Aim to log in early to avoid being held in a queue (if the event is reaching thousands of people, this is a reality). Maybe open up a separate browser window to follow the event hashtag and the event organizer on Twitter, in case you want to follow along with the social activity or ask a question outside of the platform.

Every platform is different, so if you haven’t used it before it’s worth checking the Help section to be sure you’ll have a smooth experience.

Be present

Activate your out-of-office auto-response and stick to it. Close down work and personal email and mute other communication channels like Slack or anything else that you think could distract you. Avoid the temptation to multi-task, unless it’s related to the event – it’s okay to watch a session and type a question to the speaker, but think twice before you join a team Zoom call while listening to the keynote speaker. You wouldn’t do this if you were at an in-person conference, so avoid doing it virtually.

Be realistic

Take regular breaks and maybe walk to the end of the garden in between sessions to imagine that you’re there in person. This could help you digest one session and go through what you’ve learned before you start the next one. Consider not watching sessions back-to-back by perhaps taking the time to engage with speakers directly after their session, or connecting with vendors who have solutions that might help solve your challenges. You can always watch the next session at another time, and this will allow you to finish one task before immediately starting another. It will also give your brain the time it needs for processing, and that’s when the big ideas sink in.

Get social

A massive part of the discovery and learning from face-to-face events comes from talking to others and connecting with like-minded peers who are going through the same challenges. If you only watch the sessions presented at a live conference and stay seated in the same session room for the duration, you miss the networking opportunities as well as the food and drinks.

It’s the same at online conferences. Networking might not come naturally, and you may prefer to dedicate 100% of your time to listen to the experts, but you’re actually missing a trick. Introvert or extrovert, online events can help you here. They do away with any social awkwardness because you don’t have to make small talk at the coffee machine. Instead, there are probably Q&A tabs you can open to start a discussion with anyone and everyone. Some platforms even offer to set up ‘hallway style’ meetings.

Use these to your advantage – take the temperature, ask the stupidest question, be the person with their hand up. It’s your time to get your questions answered, and by doing this, it will get your ideas flowing. If the conference offers a way to connect with attendees, speakers and vendors, use it to get acquainted. Everyone else is probably keen for a little interaction too. Remember we’re all in this together.

Go beyond

Consider what you’d be doing if you were at the conference in-person. You can probably simulate that experience and do some things even better than you might have done before. Because the event has gone virtual, you’re likely to be joined by hundreds, even thousands, more peers than you would have been if you were restricted to a conference center or a hotel ballroom.

Think of the possibilities! What’s your company’s policy on attending online conferences – do they even have one? Have you had to put in a request to attend this virtual event? If not, you might likely be stuck being in two places at once (refer to ‘Be present’). Can you still submit expenses?

Get in event mode. Beyond making the most of the more natural ways to connect with people, consider ordering breakfast pastries from a local bakery, or a lunch panini from a local deli. Helping to support local companies who aren’t currently serving business people in your local area feels like the right thing to do. Whatever you treat yourself to during the virtual event will cost much less than what you would have spent on the train/flight, hotel room and evening meals if time had stood still and it was still 2019.

More importantly, it will make the event an event, not a few hours staring at a screen.

Enjoy yourself

I’ll end this post on a high and encourage you to remember what life is all about: fulfilment, helping others, gratitude and discovery. Remember that there are faces behind the screens; people are making this happen, humans invented the tech, and everyone has feelings. Be thankful of others, as well as yourself and the rest will come.

Commentary Competition

Enjoyed the topic? Have a relevant anecdote? Disagree with the author? Leave your two cents on this post in the comments below, and our favourite response will win a $50 Amazon gift card. The competition closes two weeks from the date of publication, and the winner will be announced in the next Simple Talk newsletter.

The post How to survive and thrive when watching online conferences appeared first on Simple Talk.

from Simple Talk https://ift.tt/2EhcB09
via

Responsible AI news from Build

Microsoft Build took place more than a month ago, but only now I’m writting about it. I’m facing the danger of being nicknamed as one famous Brazilian formula 1 pilot. Internal Brazilian joke.

Build is a historical mark among community events. It implemented many new techniques to make an online event closer to an in-person event – and the result was great. Three days, almost non-stop, full of technical knowledge.

It’s impossible to write one single post about Build without missing a lot of information. Even me, crazy enough to follow the event for the entire three days, will miss details from sessions I haven’t attended.

You can watch the sessions on-demand on this link: https://mybuild.microsoft.com/ but I hope with the help of the many posts I will write about Build you will be able to make good choices of sessions you want to watch.

Responsible AI

It’s very tempting the idea of having an AI which makes the decisions for you and you are not responsible for the decisions. However, this can lead to the worst Ayn Rand scenarios, where no one is responsible for anything, even evolving to a Skynet scenario.

We need to be responsible. If the AI denies a loan, an insurance coverage or payment, we need to be responsible for the result.

Notwithstanding, we also need to remember AI depends on data. If we have a small set of data about some groups of our society because our society isn’t fair, this can lead some bad AI models to have a “racist” behaviour.

In order to make us responsible, we need tools which are able to “open” AI models and explain to us why the model made one decision instead of another.

Let’s check some tools and source of information to achieve this.

FairLearn

https://fairlearn.github.io/

This website provides tools to measure the fairness of your AI models using metrics about your models. You can read a step-by-step article about how to implement fairlearn API’s with Azure Machine Learning here: https://opendatascience.com/how-to-assess-ai-systems-fairness-and-mitigate-any-observed-unfairness-issues/

There is also a video in Channel 9 about how to do the same thing: https://channel9.msdn.com/Shows/AI-Show/How-to-Test-Models-for-Fairness-with-Fairlearn-Deep-Dive?term=build2020%20responsibleml&lang-en=true

There are also articles on the Microsoft Research blog about responsible AI: https://www.microsoft.com/en-us/research/blog/machine-learning-for-fair-decisions/

In order to go deeper in the concept, there is this study made by Microsoft Research, the University of Maryland and the Carnegie Mellon University: https://arxiv.org/pdf/1812.05239.pdf

This other study explains the math behind an approach for fair classification: http://proceedings.mlr.press/v80/agarwal18a/agarwal18a.pdf

More in deep details and many links on this document: https://sites.google.com/view/fairness-tutorial

Webminar about Develop AI Responsibly: https://info.microsoft.com/ww-ondemand-develop-ai-responsibly.html?lcid=en-us

InterpretML

InterpretML is a framework used to explain decisions made by AI models, allowing to check if the decision is fair.

The framework is available on github: https://t.co/1q4ft5aToM?amp=1

Here you can watch a video about how to use InterpretML: https://channel9.msdn.com/Shows/AI-Show/How-to-Explain-Models-with-IntepretML-Deep-Dive?term=build2020%20responsibleml&lang-en=true

Here you can watch a video about a use case of InterpretML in an airline company: https://channel9.msdn.com/Shows/AI-Show/InterpretML-in-Practice-at-Scandinavian-Airlines?term=build2020%20responsibleml&lang-en=true

DICE

Dice is a Microsoft Research project used to make counterfactual explanations of ML model decisions.

For example, let’s say you had a loan rejected. Dice will provide you alternative possibilities which would result in the loan approval, such as “You would have received the loan if your income was higher by $10k”.

You can read more about DICE and get the framework on this link: https://www.microsoft.com/en-us/research/project/dice/

Responsible AI resources from Microsoft

This link is a central place for responsible AI with microsoft technology, with many videos, articles, frameworks and guidelines about how to ensure a responsible AI while building your models

https://www.microsoft.com/en-us/ai/responsible-ai-resources?activetab=pivot1%3aprimaryr4

Conclusion

Considering all these tools to ensure the quality of result from AI, movie characters such as HAL or Skynet are way far from reality than before.

The post Responsible AI news from Build appeared first on Simple Talk.

from Simple Talk https://ift.tt/2ExmdTU
via

Understanding digital transformation & how it transcends to driving organizational growth

Introduction

Digital transformation is the process of undergoing multiple internal and external changes by an organization leveraging digital technologies to become more agile, optimized and efficient in its business operations. Dynamic IT environment, the urge to leverage emerging technologies, and the unpreparedness to function at scale in throes of the global pandemic are driving increased spend on cloud-based applications and further pushing the case for digital transformation. MarketsandMarkets data suggests, digital transformation market is envisioned to witness a CAGR of 18.1% to reach $665 billion by 2023 from $290 billion in 2018^[1]. IDC report forecasts, global spending on public cloud services and related infrastructure is anticipated to hit $500 billion in 2023, from $229 billion in 2019, a CAGR of 22.3%^[2]. All these numbers indicate a strong surge in cloud and intelligent digital technology adoption as more and more companies embrace digital transformation to become agile and stay market relevant. This article provides a deep dive into what is ‘Digital Transformation’ and how it transcends to driving organizational growth.

Digital transformation is about being agile and responding to the current changes & preparing for future changes leveraging digital technologies and its potential to enhance human experience and capabilities in all spheres of business. It is about cultivating a mindset of continuous hustle, continuous optimization, holistic growth and focused efforts on what people need(Customers, partners, employees).Let’s try and break this definition of digital transformation and look at each of the parts individually to understand it better:

• Responding to Current Changes
• Preparing for Future changes and building innovative solutions
• Enhance Human Experience & capabilities in all spheres of business

Responding to current changes leveraging digital technologies

This part focusses on realizing that an organization needs to have digitally enhanced customer-driven business strategy. As a first step, IT and business goals need to align to channelize resources in creation of digital products and experiences which can drive growth in current markets and products. From an organization perspective, the focus is to transform to optimize operations, enhance customer-centricity, manage risk, find new use cases to deploy digital technology and increase revenues. Digitization and Digitalization are all part of this strategy.

The digital transformation wave is pervasive across industries & geographies . Digital technologies provide immense opportunities for organizations to evaluate possible use cases to optimize business operations and improve bottom line as part of their digital strategy. Now, the question arises-What can organizations do to attain Digital Transformation in responding to current changes?

As far as responding to current changes leveraging digital technology is concerned, the focus is to significantly improve the customer experience, act on insights derived from real-time and predictive analytics, increase productivity, cross- enterprise collaboration, automation of mundane tasks and unleash business value in a non-disruptive way. The organization needs to identify and re-define relevant business processes which can deliver following outcomes leveraging digital technologies:-

Deliver customer centricity and experience

Delivering Customer centricity & experience at every touch point by creating innovative solutions that deliver unique experiences by focusing on personalization, convenience, ease of doing transaction, value for money or privacy preserving practices to differentiate. For example, Netflix, Amazon & Spotify’s predictive hyper-personalized experience embedded in their recommendation systems is a classic case of delivering customer delight and unique experience customized to each individual customer.

Real time analytics

Gone are the days when business users waited for batch jobs to update numbers on predetermined frequency, today real time analytics has become norm as it provides immense potential to respond to changes, prepare for the unexpected, make smarter decisions, optimize operations and tasks in turn reducing cycle time, and innovate with agility & speed. IoT sensors and devices are driving the case further for real time actionable insights through streaming analytics. Grandviewresearch^[3] states ”The global streaming analytics market size was valued at USD 6.32 billion in 2018 and is expected to grow at a compound annual growth rate (CAGR) of 28.9% from 2019 to 2025.” For example, Swiss Federal railway^[4] replaced their old steam trains with technologically advanced fleet and started using proactive asset maintenance by using SAP HANA streaming analytics to determine which loads should be decreased and this information flows to other systems which automatically turn off heaters in train cars and railroad switches, helping the company in optimizing operations and reducing costs.

Predictive analytics

Embedding predictive analytics can help organizations focus on proactive business process management than reactive. The ability to predict future outcomes and prescribe what needs to be done in supply chain efficiency, cyber security and disaster management upfront can turn into game changing competitive advantage. For example, in oil and gas industry, predictive analytics is helping companies improve operational efficiency by proactive predictive maintenance, demand forecasting, energy trading, risk management and optimization.

Robotic process automation

Automating voluminous and repetitive tasks can help in decreasing manual effort on non-core activities which will help channel the saved time & human effort in more productive and innovative activities. Automated shared services, augmented intelligence and smart automation are great ways to put digital technologies to drive operational efficiency , cost reduction, revenue growth and competitiveness. Returns processing is a common use case for RPA that we have all seen. RPA helps companies manage returned products by automating manual tasks such as receipt of return confirmation, inventory update , billing system update and issuing payment adjustment to the customer.

Preparing for future changes and building innovative solutions leveraging digital technologies

Preempting and preparing for future market changes focusses on trying to aggressively decipher disruptive potential of digital initiatives, technologies, newer business models, new opportunities to extend or diversify into newer markets. Redefining existing marketplace and creating new market to tap into first mover advantage and innovate constantly to ensure they remain fast moving target for their competition. Here , the focus is to tap into entirely new revenue streams with new business models, markets revolving around information and services leaping far beyond optimization aspect.

Quest for creating newer innovative business models to thrive in today’s digital world is a major challenge for executives. The value proposition of newer business models, the required core competencies and skillsets, forging strategic partnerships , understanding market forces, generating shareholder value along with providing tangible benefits to target customers is easier said than done. So, how does an organization decide on which newer business models are promising for future revenue growth? Well, the answer is “it depends” and varies from company to company. However, reimagining business models can be done in following ways:

Venture into new markets and industry

Look at your products, core assets, technological capabilities, data , customers, Intellectual property that you have built over numerous years of existence. Explore options and decipher opportunities to leverage these assets to enter or disrupt other industries who may/may not be part of your existing value chain. Disrupt entire value chain by thinking out of the box and reshaping the way your products and services reach their target audience. UPS is a perfect example of leveraging intelligent technology for newer business solutions. UPS^[5] is growing from delivery company to a niche player offering 3D printing of critical and quality auto spare parts to its customers and tapping into un-catered need of printing one of a kind items and prototypes.

Harness the power of your ecosystem

Collaborating and integrating with other businesses in your ecosystem to provide a seamless experience to common customer. Community based sharing of underutilized resources in peer network is another innovative way to redefine business models using digital platforms and technologies. The use case will differ from organization to organization but the potential to deliver newer innovative solutions without boundaries can surely disrupt the isolated players. For example, partnership between Concur, Nokia, Hertz and Mojio is taking connected vehicle to a different level by integrating car rental to parking and fueling location and payment, to integrated navigation and expense management for business travelers^[6].

Power of your online community

Monetize the power of your online community of users and customers by mining their opinions, deriving insights, driving member engagement for research initiatives for third party organizations. As a bonus, you may also drive open innovation initiatives to find newer business models, use cases from the larger community. There is plethora of opportunities to unlock the potential of virtual communities and drive next disruptive product development by building on digital technologies. For example, Facebook uses hackathons to generate newer ideas and fosters a product innovation centered culture built on open innovation inside and outside the company. Another example is SAP Ariba network, where buyers and sellers collaborate and discuss new business opportunities.^[7]

Outcome based business models

Create and repurpose your business model on the final tangible outcome that your organization delivers instead of mere products and services. Build capabilities to position your business as an enabler of the larger outcome and not just the product or service sold. For example, Zipcar provides on demand IoT enabled cars and trucks .Customers pay for subscription which includes gas, insurance and on demand access to cars and trucks. Their outcome-based business model uses IoT sensors to drive innovation in car rental business. Another classic example of outcome- based business model is Amazon web services which provide reliable, scalable , high availability cloud services.

Enhancing human experience & capabilities in all spheres of business

Enhancing human experience & capabilities in all spheres of business, is indeed the most important part of the digital transformation narrative, if a change isn’t enhancing customer experience or complementing employee expertise it may not be a transformative in true sense. It is not just these digital intelligent technologies alone that propel organizations to true transformation, but it is their interaction with humans which deem these emerging technologies game changing and revolutionary. As organizations deploy more digital technologies like Machine Learning, cloud, security, analytics and IoT ; the workforce will need re-skilling and retooling to bridge the skill gap and drive employee engagement.

So, now that we know how we can achieve digital transformation leveraging digital technologies in current business scenario and how to create newer business models , now the question arises- what can we do to ensure the digital technology enhances the human experience?

To answer this, we need to look at intelligent process automation, use of AI in automating mundane repetitive tasks and channel employees’ time and effort in more value adding, productive tasks. Not only automation, but intelligent technologies such as intelligent analytics, IoT, voice enabled devices, virtual reality , smart wearables, integrated workflows , collaboration platforms will enable faster decision making and more transparency. And in turn drive employee reskilling and which will enhance employee engagement and productivity.

Conclusion

In nutshell, the Digital transformation is an amalgamation of all these 3 parts and requires a digital transformation strategy that looks at goals, current situation, business context and chalks out a clear roadmap of how to move forward on all the three frontiers. The possibilities are endless, the opportunities are boundless and perils of lagging behind and failing to leverage these digital technologies are grave; the onus of acting on these key focus areas and making most out of the intelligent digital technologies and redefine markets and industries lies with the executives.

Hope this blog gave you some insight into understanding Digital transformation and how you can incorporate it in your organization to become a digitally transformed market leader.

References

[1] https://ift.tt/34nJkvo

[2]https://ift.tt/3l8xBX9

[3] https://ift.tt/31idfmH

[4] https://ift.tt/31idfTJ

[5]https://ift.tt/3j2Z9M2.

[6] https://ift.tt/2QfBmfq

[7] https://ift.tt/2rtSJf9

 

The post Understanding digital transformation & how it transcends to driving organizational growth appeared first on Simple Talk.

from Simple Talk https://ift.tt/3l6WnHh
via

Storage 101: The Future of Storage

The series so far:

1. Storage 101: Welcome to the Wonderful World of Storage
2. Storage 101: The Language of Storage
3. Storage 101: Understanding the Hard-Disk Drive 
4. Storage 101: Understanding the NAND Flash Solid State Drive
5. Storage 101: Data Center Storage Configurations
6. Storage 101: Modern Storage Technologies
7. Storage 101: Convergence and Composability 
8. Storage 101: Cloud Storage
9. Storage 101: Data Security and Privacy 
10. Storage 101: The Future of Storage

An IDC report published in November 2018 predicted that the world’s data would grow to 175 zettabytes by the year 2025. For those unaccustomed to such amounts, a zettabyte is about 1,000 exabytes, which comes to one billion terabytes or one trillion gigabytes. Given our current trajectory, we’ll likely see those predictions come true. Even if we fall short, there will still be a heap load of data.

Current storage technologies are going to have a tough time keeping up. They’re already having a tough time keeping up. With the explosion of mobile devices, followed by the influx of the Internet of Things (IoT), more data than ever is being generated—by people, by applications, by machines. The only way to derive meaning from all that data is to develop innovative high-performing, high capacity storage solutions.

Figure 1. Data’s exponential growth (image by geralt)

Scientists are pioneering storage solutions that can support our data loads into the future. To this end, they’re searching for ways to improve NAND flash and storage class memory, while experimenting with new storage technologies. In this article—the last in my series on storage—I provide an overview of many of these efforts to give you a sense of what we might expect in the near and, with any luck, not-too-distant future.

What’s up with NAND flash?

NAND flash adoption has significant data center market share, offering substantially better performance and durability than what hard-disk drives (HDDs) are physically capable of ever achieving. As NAND’s popularity has increased, along with its densities, prices have steadily dropped, making it a more viable storage option than ever.

Yet even with these improvements, they’re not enough to meet the demands of many of today’s data volumes and workloads, which is why vendors are working hard to make solid-state drives (SSDs) that can deliver better performance and greater densities while minimizing the cost-per-GB.

The primary strategy for doing so is adding more bits per cell, more layers per chip, or a combination of both. Flash SSDs have gone from one bit per cell to two bits and then three. Now we have quad-level cell (QLC) SSDs, which squeeze four bits into each cell. Initially, QLC flash primarily targeted PCs, but that’s starting to change, with some vendors now offering QLC storage for the data center.

More bits per cell increases the need for error correction, slowing program/erase (P/E) cycles. The additional bits also decrease endurance as cells become more labile. Until significant advances are made in P/E processes such as garbage collection, enterprise QLC flash will be limited to read-intensive workloads. In the meantime, vendors are pushing ahead with more bits per cell, even developing penta-level cell (PLC) SSDs that boast five bits per cell.

At some point, adding more bits per cell will no longer be practical, which is why vendors are also adding more layers to their NAND chips, a technology referred to as 3D NAND. In this type of chip, memory cells are stacked into vertical layers to increase capacity. The first 3D NAND chips had 32 layers. Many vendors now offer SSDs with 96 layers.

In addition, several vendors are ramping up production on 128-layer SSDs, with 256-layer devices on the horizon. Devices featuring 500 and even 800 layers or more are forecast. But additional layers mean thinner materials, amplifying manufacturing challenges and costs. The cost-per-GB is unlikely to decline as quickly as it has been without novel technological advances.

Who’s invading the flash space?

While vendors continue to enhance their NAND flash offerings, some are also investing in technologies that could eventually replace flash or be used in conjunction with flash to create a hybrid solution. One of these is Intel’s Optane DC SSD, which is based on the 3D XPoint architecture, a storage-class memory (SCM) technology developed by Intel in partnership with Micron.

The Optane DC SSD provides greater throughput and lower latency than a traditional flash SSD, including Intel’s own line of enterprise flash storage. IBM is now working on its second generation of the Optane DC SSD, offering hints that it might nearly double the speed of its first-gen implementation.

Figure 2. Moving into flash territory (image by Fotocitizen)

Not to be outdone, Samsung now offers its own alternative to traditional NAND flash—the Z-SSD drive (or Z-NAND). Although the Z-SSD is based on NAND technologies, it offers a unique circuit design and controller that delivers much better performance. In fact, the Z-SSD is often described as an SCM device and is considered Samsung’s answer to Intel’s Optane DC SSD.

Micron has also released an SSD built on the XPoint architecture—the X100 NVMe SSD. Both Micron and Samsung appear to be planning their next generation of flash alternatives. But they’ve released few details about the devices or how they’ll perform.

In the meantime, Kioxia (formerly Toshiba Memory) is working on its own NAND flash alternative, Twin BiCs FLASH, which the company describes as the “world’s first three-dimensional (3D) semicircular split-gate flash memory cell structure.” That’s quite the mouthful and certainly sounds intriguing. However, the project is still in research and development and will likely not see the light of day for some time to come.

It’s uncertain at this point what the future looks like for NAND flash alternatives such as those from Intel, Micron, Samsung, and Kioxia. Much will depend on how traditional NAND flash evolves and the affordability of these new devices over the long-term. With workload and data demands increasing, organizations will continue to look for whatever solutions can effectively balance performance and capacity against endurance and cost.

Where does storage class memory fit in?

In the last couple years, storage class memory (SCM) has inspired many headlines, especially with IBM’s recent release of the first Optane DC persistent memory modules (PMMs).The modules plug into standard dual in-line memory module (DIMM) slots, allowing the PMMs to connect directly to the server’s memory space. The Optane DC modules represent a big step forward toward the vision of a new storage tier that sits between traditional dynamic RAM (DRAM) and NAND flash storage to support demanding enterprise workloads.

Intel’s Optane DC modules are typically referred to as a type of phase-change memory (PCM)—“typically” because the company’s messaging has been somewhat mixed around this issue and they are sometimes considered to be a type of resistive RAM. However, the consensus is that the Optane DC modules fit neatly into the PCM category.

Phase-change memory is a type of nonvolatile memory that stores data by rapidly changing a material between amorphous and crystalline states. Phase-change memory offers much faster performance and lower latency than NAND flash and has the potential of delivering greater endurance. On the other hand, PCM is also much more expensive.

But PCM is not the only SCM effort under development. Scientists are actively researching other technologies that they believe can also serve as a bridge between DRAM and flash storage. One of these is resistive RAM (RRAM or ReRAM), another type of nonvolatile memory that promises significantly greater performance than NAND flash, with speeds approaching those of DRAM.

Resistive RAM works by applying different voltage levels to a material in order to switch its resistance from one state to another. Compared to NAND flash, RRAM offers much better performance and higher endurance while consuming less power. In fact, the technology shows so much promise that it has been proposed as a possible replacement for both NAND flash and DRAM.

Another nonvolatile memory technology that shows promise is ferroelectric memory (FRAM or FeRAM), which is built on a ferroelectric capacitor architecture that incorporates a mechanism for controlling polarities. Ferroelectric memory offers high read and write speeds, low power consumption, and high endurance. But in its current form, it has a very low density and its processing costs are high.

Nanotube RAM (NRAM) is another nonvolatile memory technology that’s being actively researched for its DRAM-like performance, low power consumption, and ability to withstand extreme environmental conditions. Nanotube RAM can also retain data far beyond NAND flash capabilities. A NRAM device is made up of tiny carbon nanotubes that are extremely strong and have conductive properties. The nanotubes sit between two electrodes through which voltage is applied to change the resistance, providing the structure for data storage.

Researchers are also focusing on Magnetic RAM (MRAM), which could potentially deliver speeds on par with static RAM (SRAM). Magnetic RAM—also called magnetoresistive RAM—is a nonvolatile memory technology that uses magnetic states to store data bits, rather than using electrical charges like other memory technologies.

Vendors are pursuing different strategies for implementing MRAM. One of the most promising is spin tunnel torque MRAM (STT-MRAM), which leverages the angular momentum in quantum mechanics to store data. The biggest challenge with MRAM, however, is its extremely low density.

All of these memory types—along with others being investigated—are in various stages of research and development. Although several vendors already offer products based on some of these technologies, today’s research is what will drive them into the future and make it possible to create a memory-storage stack in which all memory is nonvolatile, profoundly changing the way we deliver applications and store data.

What does the future hold?

The memory technologies I’ve discussed so far are mostly works in progress, with vendors looking for ways to make them more practical and profitable beyond a handful of small niche use cases. But researchers are also looking further into the future, working on technologies that are still in their infancy or have been around for a while but are now being infused with new efforts.

One area of research that’s caught the industry’s imagination is silica glass, which can be used to store data much like the crystals that taught Superman about his Krypton roots. This idea of silica glass got its boost in 2013 from researchers at the University of Southampton, who demonstrated storing a 300 KB text file in fused glass.

The storage medium, referred to as 5D memory crystal, or 5D storage, relies on superfast femtosecond laser technology, like that used for refractive surgery. The laser etches microscopic nanogratings into the glass to provide the data bit structure. A special technique is then used to retrieve the data, taking advantage of the light’s polarization and intensity.

According to the researchers, a 25-mm silica disk could store as much as 360 TB of data, sustain temperatures up to 190 degrees Celsius, and remain viable for over 13 billion years, making today’s storage media seem like cardboard cutouts. In fact, 5D storage has already received a fair share of notoriety. A silica disk storing Isaac Asimov’s Foundation series now orbits the sun, sitting inside Elon Musk’s cherry red Tesla Roadster, which itself sits onboard the Falcon Heavy SpaceX rocket.

Microsoft was so impressed with the 5D storage technology that it has launched its own initiative, dubbed Project Silica, whose stated goal is to develop the “first-ever storage technology designed and built from the media up, for the cloud.” Project Silica uses femtosecond lasers to write data into quartz glass, the same process used for 5D storage. As its first proof of concept, Microsoft teamed up with Warner Bros. to store and retrieve the entire 1978 Superman movie on a piece of glass about the size of a drink coaster.

Another innovative approach to data storage is racetrack memory, which was first proposed by IBM researchers in 2008. Racetrack memory applies electrical current to nanowires to create domain walls with opposite magnetic regions between them (thus the racetrack concept). The domain walls and their regions provide a structure for efficiently storing data. IBM hopes that racetrack technology might eventually yield a nonvolatile, solid-state storage device that can hold 100 times more data than current technologies at a lower cost-per-GB.

Other researchers are pursuing a different approach to racetrack memory, leveraging the inherent properties in skyrmions, which are microscopic swirls found in certain magnetic materials. Skyrmions work in conjunction with anti-skyrmions to create opposing magnetic swirls that can be used to create a three-dimensional structure for hosting digital data. Skyrmion-based storage requires very little current and has the potential for storing large quantities of data while delivering high-speed performance.

Scientists are also researching the potential of storing data at the molecular level. One of the most publicized approaches is DNA, in which data is encoded directly into the genetic material. Corporate, university, and government researchers are actively pursuing DNA’s potential for persisting data. DNA can store massive amounts of information, is millions of times more efficient than anything we have today, requires almost no maintenance, and can endure for many millennia.

Figure 3. Betting on DNA storage (image by geralt)

The challenge with DNA storage, however, is that it’s error-prone and expensive to produce. To address these issues, scientist have been experimenting with multiple solutions. For example, researchers at the University of Texas at Austin have come up with error-correcting algorithms that help compensate for the high rate of errors. Using synthetic DNA, they have successfully stored the entire book The Wizard of Oz, translated into Esperanto. But this is nothing compared to DNA’s true potential. As many have claimed, DNA could make it possible to store the entire internet in a shoe box.

Despite the enthusiasm around DNA storage, researchers are also investigating different molecular storage techniques, using molecules that are smaller than DNA and other long-chain polymers. The big advantage here is that smaller molecules can be cheaper and easier to produce, and they have the potential for storing more data. If that’s not small enough, scientists are also researching single-atom data storage, with each bit stored in an individual atom. So far, I’ve come across no discussions about going smaller.

Where do we go from here?

If technologies such as molecular storage and silica glass storage can be manufactured in a way that is both efficient and cheap, we’ll be better prepared to handle all the data that’s expected in the years to come. But we have a long way to go before we get there, and until then, we’ll have to rely on the advancements being made with NAND flash and its alternatives, as well as with SCM. What we’ll do with all that data once we figure out how to store is another matter altogether. In terms of storage, however, the sky is indeed the limit.

 

The post Storage 101: The Future of Storage appeared first on Simple Talk.

from Simple Talk https://ift.tt/3hjDWwH
via