tap tap tap… is this thing on?
I have just migrated from DasBlog => WordPress, it has been a bumpy ride due to my ineptitude in both DasBlog and WordPress, bear with me whilst I iron out the bumps!
I had some pain installing Pscx on Windows 7 some of the instructions that came in the download did not work for me and other instructions assumed knowledge. The following explains what worked for me.
My environment details:
OS: Win7 Home Premium 32bit // echo (Get-WmiObject Win32_OperatingSystem).Caption $ENV:Processor_Architecture
Powershell Version: 2.0 // use get-host to check this
1) Ignore the instructions that come with Pscx download
2) After downloading Pscx-2.0.0.1.zip, right click on the file and UnBlock the file, then extract the contents.
3) Copy Pscx folder to: C:\Windows\System32\WindowsPowerShell\v1.0\Modules
4) Open a powershell prompt (powershell.exe) enter: set-executionpolicy unrestricted
5) Enter command: Import-Module Pscx
The instructions above worked for me, if they do not work for you please post a comment.
I have been trying to deepen my understanding of IoC (Inversion of Control) or “Dependency Injection” as Martin Fowler prefers.
For the first time I heard a really good explanation of what IoC is and why it is useful. This was during a podcast with Scott Hanselmann whilst he was discussing MVCTurbine in his Hanselmiutes podcast.
Scott described the process of creating a new MVC app: After you have fired up Visual Studio and clicked file > new > MVC project you must then go about the business of creating a controller. Once that is done you need to create a view and then you can fire up your app and see a web page. Now you need to start adding some infrastructure to your controller to give it the ability do things like logging. At this point you need to make a decision about where the responsibility or ‘control’ will be placed for getting the logging implementation. Now logging is a great example, nearly all applications need to log information so it is likely that you will be making use of an existing library, it is also likely that logging is going to be performed by most handlers. If you opt to make it the responsibility of the controller to locate and instantiate your logging library then all controllers will need the code that does this. If instead you opt to invert the control of this location and
instatiation then you need a mechanism to do this… enter Dependency Injection. Instead of adding code to your controller that locates and instantiates a logger you place a dependancy into your controller on an interface that describes the interraction with your logger. Then you pick a container like Autofac or Windsor and configure it so that at runtime it knows how to instantiate the implementation that you want to use for logging.
With IoC when you look at the code of the controller class all you see is a public property of the logger interface type:
public ILog Logger{ get; set; }
Unless you first understand that the dependancy will be injected at runtime the code looks a bit weird and like it couldn’t possibly work because it is not obvious how the implementation will be located and instantiated. When your application starts you must add some code that configures your chosen IoC container so that it knows how to inject the logging implementation into your MVC controller at runtime.
Greg Young has coded up a sample CQRS app with the aim of creating the simplest possible implementation.
I have been looking through the code to try and better understand CQRS.
Greg Young’s Blog
Download the Sample App Here
The diagram above is based on stepping through the code after the user submits a new Inventory Item.
I’m still not sure I fully understand things but it is starting to become clearer.
This article on StorageSearch.com shows an interesting trend and prediction in the pricing change between DRAM and NAND Flash, as you can see from the chart both are dropping but NAND is dropping faster.
What acutally happened was that NAND prices rose, likely due to demand from companies like Apple, below the first chart shows the price of 8GB MLC NAND Flash from 2007 to today. These modules are commonly used in smaller capacity 2.5″ solid state disk drives. Dram prices also rose as depicted in the second chart below.
MLC Flash 8Gb Module Pricing 2007 – Today
Dram Pricing DDR2 1Gb Module 2007 – Today
There is currently $$$ being spent in building new manufacturing facilities to produce NAND flash so unless new sources of demand appear prices will start to fall again soon. So what does this mean if you are currently designing an enterprise solution that will be deployed to production in the next 12 months and be used for up to the the next 10 years? Well it depends on your application but if it makes significant use of a relational database management system or RDBMS then you can expect a huge decrease in the cost of delivering IOPS.
Calculating IOPS cost
Today you can purchase a FusionIO card for ~$7500 that offers 320GB of storage and a R/W 100k/140k IOPS, a single 15k 450GB SAS drive costs around $300 and will deliver ~175 IOPS to match the mixed read/write performance of the card you would need to purchase 300 drives, enclosures and controllers, a rough calculation:
15k Disks – 300 x $300 = $90,000
Enclosures – 10 x $5000 = $50,000
Total = $140,000
RAID IOPS Calculator
The problem with the FusionIO solution is that it does not scale easily each card must be installed into a Pci-E slot in a server, most 1U servers will only have one slot. It is also directly attached to the server, when using a SAN storage can be shared amongst servers. If on top of this you are taking advantage of virtualisation it is possible to move VM’s from one physical host to another in realtime and your server will remain online and suffer a small reduction in performance for a few minutes. This makes it possible to take physical machines offline for maintenance without affective the availablity of your application. FusionIO does not offer fault tolerance if the physical machine that the card is installed in fails then you lose access to that storage until the machine becomes available again. In this situation you must have a mirror machine available with a recent copy of your data. But this is still a good value solution at $15,000 vs $140,000.
The water is a little bit murky
It is still not clear exactly how vendors are going to deliver the level of IOPS performance offerd by FusionIO in a SAN based format. The problem seems to center around the speed of interfaces like iSCSI and the bandwidth and latency of gigabit ethernet. Although the storage devices already exist there is a lot of work needed to improve the pipeline between hosts and SAN. That said this is a solvable problem, it’s not “if” but “when”.
The impact on database performance
RDBMS’s like Sql Server, MySql and Oracle are going to get a huge performance boost as NAND Flash based storage systems come into use, typically IO has been the bottleneck in any database based application in future this will likely change to CPU. Oracle and Sql Server are both licensed per CPU/Core so when building applications this should be taken into account. Any operation that requires a significant amount of CPU time should not be perfromed by the database unless absolutely necessary. Databases should just be used for what they were originaly built and that is to persist data and allow it to be retrieved. The need for inline caches should be reduced due to this performance boost.
I have seen a few posts on the NServiceBus yahoo groups mailing list and a few tweets asking about how best to monitor NServiceBus. This is a problem I have been researching over the last few weeks, here is ths solution I came up with for triggering alerts when too many messages are in a queue.
Performance Counters
MSMQ Exposes several performance counters out of the box, these can be accessed via WMI. There are a few caveats, only active queues or queues with messages will have performance counters see here for details.
Triggering Alerts
You could write your own WMI scripts to monitor these performance counters and trigger alerts, another option is to use a monitoring tool that has the ability to setup WMI monitors. OpManager is such a tool, in a couple of minutes you can configure a new monitor that will trigger emails or SMS’s to be sent if a certain threshold is reached.
Improving NServiceBus
I don’t know enough about the innner workings of NServiceBus to make an educates assement of how we could create an NServiceBus monitoring tool but I’ll try anyway.
Based on these facts:
- Every NServceBus Service knows about the Queues it is configured to use for sending / recieving messages.
- It should be simple to make an NServiceBus service discoverable.
Would it not be possible to create a generic NServiceBus.Monitor.exe with a provider model that would allow plugins to be written for any monitoring tool / protocol?
Edit:
Found this
http://monitorwang.codeplex.com/
I’m digging into SNMP (Simple Network Management Protocol) to see what it offers in terms of application / server monitoring.
The specific scenario I have in mind is the monitoring of queues in MSMQ for our NServiceBus windows services. We need email alerts to be triggered if a message is moved to the error queue or if the number of items in a queue exceeds a certain threshold.
I’m working on a Windows 7 x64 laptop so the first thing I had to do was install the SNMP service (Under CP > Programs and Features).
After that was installed I had to restart the services applet and right click > Properties > Security on the SNMP Service and create a “public” community.
After this I had to download a command line tool Net-SNMP for issuing SNMP queries to test all was well.
I opted for the basic installation and uncheck all the other options.
Open a new command prompt so that it picks up the PATH variable added during installation then run the following command
snmptranslate -IR -Td IF-MIB::linkDown
See the README.txt for the correct response but if you see no errors you are in good shape.
Next download SnmpTools Follow the instructions. I opted to install the 32bit version using the regagentWow6432.reg
Restart the Microsoft SNMP Service then run the command:
snmpget -v1 -c public localhost 1.3.6.1.4.1.15
If all is well you should see:
SNMPv2-SMI::enterprises.15 = STRING: “snmptools by erwan.l@free.fr”
That’s all for now but I will update this post again.
As Promised Part 2
I have made a little more progress accessing the MSMQ Performance counters
I have added the following two lines to the Counters.ini file and restarted the SNMP Service:
[1.3.6.1.4.1.15.5]
counter=MSMQ Queue\Messages in Queue\acerbotic\private$\distributordatabus
Now when I issue the following snmpget query from the command line:
snmpget -v1 -c public localhost 1.3.6.1.4.1.15.5
I see the response:
SNMPv2-SMI::enterprises.15.5 = INTEGER: 2
Thus indicating there are two messages in the private$\distributordatabus queue on my laptop.
The End Game
At time of writing it seems that in order to allow our SNMP monitoring software to access this OID I need to supply it with a MIB file.
The story continues…
I have found this page that talks about using a MIB editor to create such a file.
Conclusion
SNMP is a great way to monitor your application, but it involves a good deal of upfront investment if you don’t already have a developer on your team who is proficient. We don’t have such a developer so have decided to use the .Net framework to expose peformance counters and use WMI to monitor those counters. If the thresholds we set are exceeded we will get alerts.
Yesterday I attended my first UK SQL Server User Group meeting.
Event web page
Topics
- SQL Server I/O – Tony Rogerson
- Understanding how Sql performs Joins with NestedLoops, Merge & Hash – Ami Levin
- DbSophic QURE – Workload Tuning – Ami Levin
- SQL Streaminsight – Allan Mitchell – twitter: @allanSQLIS
I was most interested in the first topic but missed the talk as I did not arrive till 18:30 and the evening started @ 17:30. I don’t finish work till 18:00 and had to get across town. Of the three talks that I caught the content and delivery was at a very high standard.
Understanding how Sql Server performs Joins with NestedLoops, Merge & Hash – Ami Levin
If you write a lot of SQL queries or are often called upon to tune SQL Server then you should learn how/why the query optimizer selects from three physical operators to perform joins and how sometime it can be caught out. This talk was given by Ami Levin who is a SQL MVP and works for DBSophic. It was really interesting and the first time I have had any understanding of what these terms that appear in the execution plan mean. One really interesting thing that Ami pointed out is that sometimes the method Sql uses to estimate the number of rows and actual number of rows will differ by a large amount, this can result in a sub optimal operator being selected to perform the join. If when tuning you spot this sort of discrepancy further investigation is warranted. The following is a scenario that Ami gave when this can occur:
SELECT *
FROM SalesLT.SalesOrderHeader SOH
INNER JOIN SalesLT.SalesOrderDetail SOD ON SOH.SalesOrderID = SOD.SalesOrderID
INNER JOIN SalesLT.Product P ON P.ProductID = SOD.ProductID
WHERE SOH.TotalDue > 1500
AND SOD.UnitPrice > 300
AND P.StandardCost > 300
So in the SQL above the optimizer will use statistics to estimate roughly how many rows it will return by looking at each table in turn and estimating how many rows each where clause will return. In this case say that each Where clause will return ~1% of the rows in a table containing 100k rows so the optimizer estimates the number of rows to be 1% of 100,000 = 1000 rows then, 1% of 1000 = 10, then 1% of 10 < 1. So the optimizer thinks 1 row or less will be in the results set. Unfortunately what the optimizer cannot know is that the where clauses are releated to the data, in this case ORDERS that cost more and have more expensive unit cost and a higher so the actual number of rows returned will be nearer to 1000, hence why sometimes the optimizer can choose a sub optimal physical operator to perform a join.
If you think the wrong operator is being chosen you can use the OPTION keyword this article discusses in more detail:
To Produce this:
================
<ProductModelData ProductModelID="7" ProductModelName="..."
ProductIDs="product id list in the product model" >
<ProductName>...</ProductName>
<ProductName>...</ProductName>
...
</ProductModelData>
You need this:
==============
SELECT ProductModelID as "@ProductModelID",
Name as "@ProductModelName",
(SELECT ProductID as "data()"
FROM SalesLT.Product
WHERE SalesLT.Product.ProductModelID =
SalesLT.ProductModel.ProductModelID
FOR XML PATH ('')) as "@ProductIDs",
(SELECT Name as "ProductName"
FROM SalesLT.Product
WHERE SalesLT.Product.ProductModelID =
SalesLT.ProductModel.ProductModelID
FOR XML PATH (''), TYPE) as "ProductNames"
FROM SalesLT.ProductModel
WHERE ProductModelID= 7 or ProductModelID=9
FOR XML PATH('ProductModelData')
The key is the ", TYPE" after the "FOR XML PATH(''), the example on MSDN is missing this key component.
First let me say I love Spotify and use it everyday at work, because I love it I feel compelled to discuss the company’s failure to captialize on their initial brilliant idea to use P2P networks as a way for people to choose the music they want to listen with either the advert subsidised or premium service.
What they got right was the business model and the user interface design. The software is easy to use, the obvious feature that is missing is improved playlist management. What I cannot get my head around is how long it has taken for new functionality to be added to the client to illustrate my point I trawled the Spotify blog for the details of releases during 2008 the list is below. Bear in mind development started in September 06′ and the first beta was released in May 07′ so by 2008 the software should have been pretty stable.
New features added in 2008 for Windows Client
- User settings can be changed.
- We mark all collaborative playlist (including your own) with a padlock icon.
- Context menus have been added for the artist title on artist and album pages. (0.2.12)
- a new start page with top lists has been added (0.2.12)
- Spotify now pauses when waking up from hibernation (0.2.13)
- A new view has been added to visualize the play queue. (0.2.14)
- Growl support has been added. (0.2.14)
- Dropping a link to a playlist, artist or an album on the play queue will cause Spotify to play future tracks from that source.
(0.2.15) - History now shows the currently playing track as well.
(0.2.15) - New streaming protocol (0.3.0)
- Proxy Server Support (0.3.3)
- Improved Buffering Algorithm (0.3.3)
- Undo for playlist operations has been added (0.3.8)
So in the course of a year there are no notable features added, there was a ton of bug fixes and load of work done to polish the UI. The big question is why, were the team too busy firefighting or doing lots of work to the infrastrcuture or were they hampered by their choice of language and tooling whatever the reason it is shocking.
Spotify’s main failures
- Not Entered the US Market / gone global
- Not solved the scalability issues
- Still missing some important labels / artists (ACDC / Oasis / Beatles etc.)
- Not delivered any meaningful features to the Windows client for 3 years (with the exception of collaborative playlists)
As things stand they have left a clear path for competitors to come into the marktet and take the lions share. The barriers to entry are deals with the record lables and the p2p technology.
If you want to send feedback to spotify email feedback@spotify.com
