Balancing resources in SQL Server

Level: 3 where 1 is noob and 5 is totally awesome
System: SQL Server

As software developer I believe in decoupling, and I'm certain that decoupling is just as important in database design as well. I really like having things in minimum chunks, and keeping them independent as possible. Independence and decoupling equals scalability. This post is based on a professional experience, which supports my belief.

At my work we are developing a billing platform, and this platform has several clients. Each client use our platform to bill a certain amount of customers. So this set the scene.

Some time ago, we had one database for each client in one instance, which is a easy pattern. But in SQL Server it can easily be an expensive anti-pattern, and it is certainly a pattern which is my last choice. The problem with this pattern, it is only effective, when the databases is equal regarding size and usage.

The buffer pool

The explanation is, there is only one buffer pool pr. instance. SQL Server is an in-memory database, but in most cases, there is not enough memory for a whole database in the memory. And if there is more databases in an instance, then it is almost certain there no room in the memory for all the bases. To solve this issue, SQL server has the buffer pool, which is a cache. When querying, SQL Server will get the result from the buffer pool. If it not available in the buffer pool, it reads from the storage to the buffer pool. If there if no room in the buffer pool, it will flush some of the buffer pool, to make room for the requested data. Everything has to go through the buffer pool.

The process of getting data from storage to the bufferpool, generates PAGEIOLATCH waits. A high count of PAGEIOLATCH waits can be a sign of much data is being loaded from disk to the buffer pool.

Let us say, we have 3 databases in one instance. They respectively supports 25.000, 150.000 and 300.000 customers. The sizes of the databases reflects the number of customer, because more customers generates more data. What will happen, is the database with the most usage, will push out the data from the other bases, out of the buffer pool. This will give the lesser bases more read from the disk, producing more PAGEIOLATCH wait, adding more latency to the lesser bases. Actually, the smallest databases will suffer the most.

The poor solution  

I'm tempted to call this, the poormans solution, but it can be very expensive. The solution is to extend the buffer pool, with more memory. But there is a physical and economical limit to this. SQL Server 2014 has a new feature called buffer pool extensions, where it is possible to extends the buffer with storage(preferable SSD drives). I guess extending the buffer pool, will only cache more of the database with most uses. Besides, we have tried this feature, and have yet to see some great results. I would really like to hear, if some has used this feature with success.

The better solution

A easy way to balance databases resourcing over an instance, is by using the resource governor feature in sql server. It works excellent, but it is really a doubled edged sword. Used wrong it can really downgrade performance. It would also require some nursing, which I prefer to keep to a minimum.

The best solution

I find the best solution is to have more instances. If I have 3 clients, I will assign an instance to each, make it thier domain. Each instance, should have hard assigned a certain amount of memory, and their databases would never be able to inflict each others performance. 

This was the solution we did end up with, and it gave us better performance overall. 

  

SQL 2014: Getting started with OLTP, OLAP, SSIS, SSAS and SSRS

As passionate developer, I come in touch with many technologies, but only a few of them has been so tough to crack as SSAS and SSRS. While learning SSIS, I felt i was learning a lot about datawarehousing, and I couldn't really imagine there could be more to it. At the same time I knew there must be more to it, as SSAS and SSRS was a part of the SQL Server suite.

Now I know that SSIS, SSAS and SSRS can be translated to data gathering, data modelling and data visualisation.

And I have written this post to give a swift explenation of SSIS, SSAS and SSRS so it might be easier for you to get started.

My dream database setup 

Starting with a greenfield project, is often difficult. There no experiences and no knowledge, thereby there is paved no road, to go by when developing the system. In my opinion, the 1st mistake which the most make at this point, is to put it all data in same database. This leads to a monolith system, which is hard to maintain and expand. The 2nd mistake is not to divide the database setup in OLTP and OLAP. This lead to analysis with heavy querying in production and passive historic data also stored in production.

OLTP is an acronym for Online Transaction Processing. In other words, it is the database, which is in the line of business. You can also call it, the database with live data. In my dream setup for at system, i would have a bunch micro services, all supported by their own database. Not necessarily relational databases, I would have the database which fitted the job best. The data in these databases, would be on a journey, which ended in a OLAP system. This journey could span from instant to the lifetime of the system.

OLAP is an acronym for Online analytical Processing. OLAP is almost always, analysed for financial reasons, and therefor it should be the last stop for data. It must not return to OLTP, because it would mean, that history could be re-written which is a problem when it comes to finance.

The process of getting data from OLTP to OLAP, would be data gathering, which is the perfect intro for next topic: SSIS.

SQL Server Integration Services (SSIS)

SSIS has become one of my favourite SQL Server tools. With few drag and drops, it is possible to setup data copy or transformation. Compared to scripting(as a developer I usually prefers scripting), it is so much faster to get something up and running, and also easier to structure. Enough praising.

Referring to OLTP about having several different databases, and SSIS being able to read from different source. It is the perfect tool to support this solution.

DQS (Data Quality Service) and MDS (Master Data Service) are often mention together with SSIS. These tool works only with SQL Server, but if you are all-in with SQL Server these tools are quite good. DQS is for correcting data, as they are gathered. MDS is sort of an authority which distribute a data model through out a system to the connected SQL Server database. It makes sure data integrity is kept.

There is other suppliers of ETL tool for SQL Server, but I only use SSIS.

SQL Server Analysis Services (SSAS)

Even thou data is arranged by stars, and it browsable through excel, it can be better. While a star is easy to imagine, it can be more difficult to imagine an cube, and in the beginning even to realize the difference. A cube is like a dimensional system, as we now it from math. Most of us can imagine 1st, 2nd, 3rd dimensional system. Some of us can even imagine a 4th dimensional system, by collapsing 3 dimensions and add a 4th dimension, so it look 2 dimensional again.

Each dim table in a star, would be a dimension in a cube, but they are still not the same. In principle you could take a star, print out all tables, put the papers on a table and then draw lines for each relation from the dimension tables to the fact table. Where a cube is arranged (as mentioned before) an a virtual dimensional system, it is really not printable, but fast for analysing/browsing.

SSAS Helps you building cubes, so that would be data modelling. It will also assist you to do some data mining. I'll will refrain to speak of data mining with SSAS, because I haven't touch that topic yet.

Working with cubes, is a way in SSAS which called dimensional mode. SSAS has two more modes: PowerPivot and Tabular. It is other ways to model data, and which one is best? It depends. 

But with there is no point in having data modelled, if you can't deliver their information in a readable way.

SQL Server Reporting Services (SSRS)

As you might have guessed SSRS is the visualisation part of the SQL Server suite. There not much to say. It easy to design reports with the editor, you can render reports to various formats and make them available from a reporting server or a share point site.

Take advantages of having SSIS, SSAS and SSRS available

SSIS, SSAS and SSRS are available from SQL Server standard, BI and Enterprise. Not using these tools, when having SQL Server licenses which includes them is as failure. Often have I seen, and have properly in past done myself, custom made reports and based on some analysis directly from the production environment. 

Nobody finds it funny to create custom reports and nobody finds it funny to maintain them. So using SSIS, SSAS and SSRS is a win-win, because it would mean full value from licenses, and developers would use less time on reports and more time on funny stuff.

Getting started 

Getting started with SSAS and SSRS, was a bit hard for me, but fortunately I did stumble in the book: Microsoft SQL Server 2014 Business Intelligence Development: Beginner’s Guide by Raza Red. It is real page turner, which is nice, if you reads a lot of books per year.

It is very comprehensive and filled examples, with good explanations, which I like a lot.