2013-12-11

It seems that cloud based products and services have a significant impact on how we design, write, debug, trace and deliver our applications. The way we think about this is not the same anymore; there might be no need to have SharePoint on-premises and SharePoint Online/O365 might be a better choice, there might be no reason to host a web application on dedicated hardware/hosting provider, but Azure could bring more benefits.

All these trends cannot be simple ignored, and it is a good thing to see how new services and offerings might be used in you applications.

Logs? So, what is wrong with logs?

From the first sign, there is nothing wrong with logs. However, there are some particular challenges related to SharePoint Online, Office365 and Azure apps.

I recently had a quick look over JavaScript API for Office and with couple mates have started “Office.js TypeScript Definitions” project at codeplex. While we were working on porting out of the box Office apps from JavaScript to TypeScript, testing and adjusting samples, I was like “Wow, great, but where to write logs?!”. The same question arise with SharePoint Online and Office 365 apps – where do we write logs?

Well, we are still able to use already existing frameworks to handle logs such as log4netNLog,Logging Application Block and the rest. The same frameworks might be used for SharePoint provider hosted apps with full control on your environment. As for the SharePoint hosted apps or JavaScript based applications for SharePoint, simple list/custom rest service could be utilized to report log activities (sure, this is not a good thing!).

On the other hand, with Office365/Azure/SharePoint apps (including JavaScript apps log reporting) some challenges arise. The million dollars question has to do with log management – “Where to write logs with “html/js” based apps for Office365, SharePoint Online or Azure apps?

This is a common question for the SharePoint Online applications, and that’s predictable. We don’t have any access to the HIVE14 folder, so there is no reason to use native SharePoint ULS logging features. Surely, we might create a simple “public” list and log all the records to that list, but this hardly scalable and manageable solution. Is that scalable? Is that useful? Is that something easy to troubleshoot? Would not think so.

Do you have the same question, don’t you? So let sum up what might be the challenges and what we might want, then.

Where to store logs?

As for Azure apps, we have web/workers roles. Hence, those VMs might and will be recreate/reprovisioned time to time, so “let’s just write to the c:/logs/my-log.log” solution doesn’t really work. Alternatively, we may go with Azure Tables or even dedicated MSSQL table. This is not bad solution, but it requires additional cost for the implementation, reporting and further maintenance.

SharePoint Online/Office 365 apps do not provide any ideas regarding logs. This is totally up to you how would you handle that.

How to access logs?

Due security reasons, there might be a need for the role based access to the logs, or at least logs must not be visible to everyone. Does everyone have access to the HIVE14? – Nope. Storing logs in the public list for the SharePoint apps might not be a good idea either as there could be some sensitive trace in the log records.

How to search logs?

Reporting features over logs seem to be an important part of every application. Due various scenarios, different level of the log record might be required, quite complex queries over log data could be needed. For example, all logs regarding particular transaction on the particular server for the particular tenant or client. Multi tenancy just adds more complexity to the log management – now we may have the same problem multiplied on the amount of customer we have.

How all this might be scaled?

Finally, scaling – how all this might be scaled? There is no ultimate answer for that question due complexity and variety of applications. The biggest challenge has to do with the multi tenancy apps so that you may have literally tons of logs per every client and just thousands of clients at the same time. Having millions log records and not be able to extract necessary data could be a significant bottleneck; you may have performance issues, you may have challenges on the data store layer, and with inefficient reporting you may miss the needed insights at all.

These seem to be not complete, but quite common routines related to logs. We simple want to write, access, search and write really lots of logs. But how to get that experience with “js/html” apps fro SharePoint Online/Office 365?

Meet cloud based logging services – Loggly, LogEntries and Splunk

It is quite surprising to see that there are cloud based services for the log management. Really, “cloud based log management services” sounds weird, but this is quite logical as these service just address the market gap.

So, what do we have out there? – Well, at least LogglyLogEntries and Splunk (I may miss something, so let me know what else?). This post will not cover any pricing overview, but you always may check out “Exploring the Insanity of Log Pricing” by Kord Campbell. It might be a bit outdated so please check the actual offering at vendor’s site. Instead, I would like to show what might be out there, and what might be interesting for us to develop a better apps for SharePoint Online or O365.

So, what do these services provide? What might be useful for us?

API

LogEntries has just awesome API set; Java, JavaScript, .NET (with NLog add-in), PHP, Python. Also, it might be quite important  that all these APIs are “official” and not developed by open source community. What else may we wish to get started? Quite enough to get an easy jump.

Loggly seems to be a few steps behind. It has REST API, but it seems to be a lack of libraries support for any kind of language. There are a few open source NLog integrations projects such as NLog.Targets.Loggly but it is not official so it could be a bit risky to use it in your project.

Anyway, as far as service has REST/js based API, the service might be used for O365/SharePoint Online apps.

Reporting and search features

Both Loggly and LogEntries have reporting features, but LogEntries seems to be far ahead with it. You may create some charts, graphs, custom dashboards, email notifications and so on – this is really huge topic and probably it is better to either write a separate post or just check out reports on your own.

Anyway, just check what kind of graphs you may have with LogEntries here – LogEntries API – Graphs or a fantastic UI here – The Logentries UI. This is the way better than log files in HIVE14 or simple SharePoint list, right?

Also, these services tend to use search engines over the log entries, so reporting and search feature are impressive and scalable. If it is not enough, you may leverage REST API to query and present your log data as you wish.

Built-in integration with services/platforms

Well, this is an interesting feature from the infrastructure perspective. For example, LogEntries provides an option to integrate already existing system with LogEntries and “redirect” logs output to the cloud storage. Simple saying, you may redirect all logs from Azure straight to LogEntries! For example, check out “Azure Support” article – wow-wow-wow, that’s incredible!

Loggly has similar “platform” support, but I have not checked it yet. Let’s give that to infrastructure related posts and people, and focus on development perspective instead.

Anyway, “platform/service” integration seems to be interesting things. Probably, we may redirect all events from windows, HIVE14 logs or other on-premises system to the clouds. Why and what for? – Well, this is another question for other post.

Anything else?

Yes, Splunk. Seems to be a huge area to explore as it is a big product with serious way to do business. If you may evaluate Loggly or LogEntries in couple of hours, then you would need probably a few days to check out Splunk. I did not manage to get any kind of “cloud” service from Splunk so please point if there is one somewhere.

Finally, I surely admit that there are lots of other possible implementation of the log management; cloud based, on-premises, custom crafted and so on. The point of this post is just to look around and evaluate possible opportunities for a better logging within “html/js” based apps for SharePoint Online/Office 365 apps – nothing more. Check out what might be used right now for the better logging, give your opinion and other suggestion – this is always appreciated.

Links

Conclusion

Cloud based approach has changed and continuously changing the way we do things. Instead of writing up a new library or setting up on-premises environment, we may consider just to consume already existing service or offerings.

Loggly and LogEntries might be a good example of such a “consumerism”. As SharePoint Online/Office 365 based apps development may have challenges related to log management, we may consider to evaluate already existing offering to avoid reinventing the wheel. Simple try these services before writing up your own “log management system” and try to think how you may write apps with “cloud based consumerism” in the mind.

 

About the author 

Anton Vishnyakov