Skip to content

Tag: tyGraph

Power BI Time Values in the Browser are Different Than Those in the Desktop

Published August 18, 2017

Have you ever had this experience? You build up your Power BI report using Power BI Desktop, and then when you publish and view the same report in the service, the date/time values are off by a set number of hours. I certainly run into this often enough that it warrants a blog post.

That you’ll notice is that this offset corresponds with the difference between your time zone and UTC (Coordinated Universal Time), the universal time standard. Now, you’ll notice that I didn’t say GMT (Greenwich Mean Time) which many people take to be the time standard, but the difference is that UTC is a time standard, and GMT is a time zone. GMT observes daylight savings time, and UTC does not – it remains constant. In my case, my time zone is Eastern, which is 4 hours behind UTC in the summer, and 5 hours behind in the winter. It is always 5 hours behind GMT. This distinction becomes important as you’ll see.

Coming back to our problem, we will see this problem if we have used the Date/Time/Timezone property in a Power BI Query (or a Power Query in Excel). When one of these column types are converted to Date/Time, the value is automatically localized. The reason that we see the different values is that when it is localized in Power BI Desktop, the machine’s time zone is used to perform the conversion. When this conversion happens in the Power BI service, UTC is used, and this results in different times being used in the browser. This difference can easily throw of any measures that depend are based on time values.

Column as Date/Time/Timezone

Same column as Date/Time

Further complicating things is the fact that DAX (Power Pivot) has no concept of Date/Time/Timezone and all columns of that type are brought into the model without the automatic conversion to local. So, how do we deal with this discrepancy?

We can’t rely on the Power BI service to automatically show the viewer the correct time for the location they happen to be in, so we need to be specific about the time zone that we’re working with. In my case, I need to do this for my tyGraph for Twitter reports, like this one for Microsoft Ignite 2017, or most recently in this session browser for Ignite. For these types of reports, I adjust the times to match the time zone of the events themselves. To do this, I use the modelling capabilities in Power BI desktop to create a new calculated column in the same table that contains the UTC based date.

In DAX, time calculations are done in decimal fractions of a day. Therefore, to create a new column named StartLocal by subtracting 4 hours from a date/time column named StartUTC in a table named Sessions, the formula is:

StartLocal = Sessions[StartUTC]) - 4/24

If the source column contains null or blank values, they will be returned as 1899-12-30 4:00 AM using the formula above, so it’s a good idea to test for this case and return nulls when appropriate. The above formula then becomes:

StartLocal = If(IsBlank(Sessions[StartUTC]),BLANK(),Sessions[StartUTC] - 4/24)

This calculation will reliably return the time in Eastern Daylight Time (UTC -4) for a column that has UTC date/time values in both the Desktop and in the service. If all subsequent time calculations are based on this column, then it is a simple matter of substituting in the appropriate UTC time offset to the calculation above to show the time in a desired time zone.

It would be nice if the report would allow us to specify a “home” time zone with which to base all conversions on. Time zone could then be a property of the report. Until such a time as that happens, this should prove a suitable, if complicated workaround.

4 Comments

The State of Analytics in SharePoint and Office 365

Published May 4, 2016

After adoption of SharePoint or Office 365, one of the first things an organization will look for is some understanding as to how the product is being adopted, and what its impact on resource allocation is. Historically, options for reporting on SharePoint have been limited at best.

The Web Analytics Service application was introduced with SharePoint 2010, and relied on a series of connected Excel workbooks and a fairly Byzantine series of staging and reporting databases. It worked so well that it was removed from the product in SharePoint 2013. The Usage logs contain a rich set of information, and they are rolled up into the Usage database, but accessing the data or persisting it beyond a short time period required a fair bit of work.

There were also third party analytical solutions, but most of these came with a hefty price tag, and they focused on page views, embedding code on a page. This approach works well enough for web pages, but it doesn’t capture everything, for example document access though the .NET API. They’re therefore not always well suited to collaborative environments.

SharePoint in Office 365 was initially devoid of analytics, but some basic reports have been creeping in in recent months. With the new administration portal going live, these reports moved from the relative obscurity of the compliance center to the brand new report center, and were augmented by some additional reports.

With the release of SharePoint 2016, and the announcements made at the  Future of SharePoint Event on May 4 2016, we can see the additional areas where analytics are being introduced into the core product. At this point, it’s a good idea to step back and have a look at the Analytics landscape as it pertains to SharePoint and Office 365.

At the moment, the analytics offerings can be grouped into 4 major categories; tenant scoped, site scoped, document scoped, and Delve Analytics. Let’s have a look at each one in turn.

Tenant scoped

The tenant scoped reports are the aforementioned reports that are now available in the new Office 365 Reporting Center.

New usage reports for SharePoint OneDrive Yammer and Skype 1

There are a number of interesting reports in here that focus primarily on the tenant as a whole. How much OneDrive space users are using, Yammer message counts, Skype meetings, emails sent and received, etc. In addition, these reports can be interacted with to show four different time periods, 7, 30, 90, and 180 days. Year over year analysis is not available.

These reports will primarily interest administrators, and it therefore makes sense that they are only available in the administration center, where administrative permissions are required to access them.

Site Scoped

Site scoped analytics contain data that is of concern to site administrators. These users are more concerned with content usage than resource allocation. These analytics features were initially announced at the Future of SharePoint event on May 4 2016, and as of this writing, have not yet rolled out.

The initial rollout will focus on content consumption, visits to the site and document views

image

SNAGHTML1eec8862

SharePoint home page with activity - 100 percent

This is welcome data to beleaguered site administrators, and it will help to identify important content, and content that maybe could be pruned. While it will be initially rolling out to SharePoint Online, the good new is that on premises users will also be able to get this through the new Analytics service application.

In a similar model to the new hybrid search, the new Analytics service application called SharePoint Insights connects to Office 365 and delivers your on premises  usage data to the service – essentially everything that is kept in the logging database. From there, the service can act on it to do interesting thing. One of those interesting things will be to deliver content based activity reports like the ones seen above.

There are a few things to take note of about site scoped analytics. They are scoped to the site, not the site collection. They do not roll up into a master report, so each site must be visited in turn (they live in the “site contents” section) to see the results. As far as I’m aware, the data is only persisted for a short time (I have only seen 7 days), so time based analysis is not possible.

Document scoped

Document scoped analytics have been in the service for some time now, and the new document library exposes them. I call them analytics, but they really are just the activity stream for a document or a folder. The do offer insight, so we’ll stick with the term.

From a “new style document library, you select the information icon on the right to open up the information pane. Part of that information is the activity stream of the document. In the example below I have selected a folder.

image

It’s a welcome addition, and it is what it is. There’s currently no way to aggregate the data or to pivot on it focused on anything either than the document/folder

Delve Analytics

Delve Analytics is a new offering from the Office team that focuses on the user. It analyzes a persons communications and schedule to provide insights into their work experience, with measures like time spent in meetings, time spent in email, work life balance, etc.

Take back your time with Delve Analytics 2

Delve analytics doesn’t really belong in a blog post about SharePoint because it doesn’t analyze any SharePoint or OneDrive data, so I’ll keep this section short. For the moment at least, it is restricted to Exchange email data as a source.

Delve Analytics requires an Office 365 E5 license or it can be purchased separately. Unlike the rest of the analytics options here, there is an extra cost associated with it.

Summary

The analytics options available in Office 365 and in SharePoint have improved drastically, but are still in their infancy. Each of the approaches are targeted at different audiences (IT Pro, site admins, content authors, individuals). This approach is bot good and bad. Tailoring an approach to an audience means that the specific audience will be satisfied, but the different approaches can become somewhat disjointed. It depends on what is necessary.

Analytics at the moment are also limited to specific time slices, if time can be sliced at all and to specific dimensions/measures. This is no problem if recent activity is the only thing of interest, but if more fine grained time slices or year-over-year analyses are needed, then the out of the box approaches will fall short.

Finally, most of the reports focus on activity, there is very little information provided about the SharePoint or Office 365 inventory.

The good news in all of this is not only that Microsoft has made analytics a priority, but that all of its features in this area use publicly available APIs. this means that there is plenty of room for third party vendors to step in to fill gaps and to provide complete analytics focused solutions. In that vein, I’m very proud to announce that my company, UnlimitedViz will soon be releasing a product, tyGraph for Office 365 to do exactly that.

 

2 Comments

Ignite 2015 Impressions

Published May 12, 2015

I don’t normally do conference summaries, but Ignite was just so big, and there was so much information that I felt the need to record my thoughts around it, and decided to share. Ignite was very much cross product, which is in line with where Microsoft seems to be headed – a focus on the function, not the tooling. With around 24,000 people in attendance, the conference, and the logistical issues that it imposed was too big for my taste, but the amount of information was excellent, and I imagine that I’ll be digesting it for some time to come. For now, here’s how I interpreted it all.

Azure and Office 365

Cloud services are killing it.

Between Azure’s Platform as a Service (PaaS) and Infrastructiure as a Service (IaaS) and Office 365’s Software as a Service (SaaS), Azure Active Directory is already sporting over 450 million active users. Azure Active Directory is what is used by Office 365, and the accounts within are otherwise known as Organizational Accounts. It’s an important metric because I believe that the Microsoft strategy is to own identity online. It makes sense when you look at what they seem to be doing.

For years, they absolutely dominated operating systems. Nothing to this day has ever really touched them on the desktop, but Apple changed the base with mobile, and developers flocked there. Google tried to do the same thing to Apple, and has been quite successful, but not fully so. While Android is in the majority in the mobile space, iOS is still quite strong, and shows no signs of diminishing. Windows isn’t really a factor in mobile, but still dominates the desktop which remains significant (about 300 million units/year), and is a factor on tablets. Microsoft got flanked by Apple and Android, and is holding the fort, but not conquering any new territory.

Microsoft now seems to be focusing on cloud services, and they don’t care what platform is being used to consume them. I think that at the core of this strategy is cloud identity – whether it is consumer (Microsoft Account) or enterprise (Azure Active Directory). With this identity strategy, Microsoft is attempting to again change the base – to outflank both Apple and Google and make the operating system almost irrelevant. Every app they’re putting out now is usually for iOS first, then Android, then Windows Phone. The new Universal app platform likely means that they will come out for Windows (desktop, phone, whatever) at initial launch with iOS, but the bottom line is that an awful lot of effort is going into supporting all platforms all the time. If the apps work well across platforms, then the choice of operating system simply becomes one of personal preference, not of features. It gets marginalized, and Microsoft owns the back end service. That’s why I think that so much effort has gone into this strategy.

Another thing that I sensed at the show was that in the past, all of the talk around identity and federation (ADFS) was about bringing your on-premises identities into the cloud to support a few new services. Now, there seems to have been a real shift, and the reason for adopting ADFS is to bring the Azure Active Directory identities back down on-premises to where legacy applications can use them. It’s a subtle shift, but discernable.

One of the more interesting product introduced into Azure recently is Logic Apps. As far as I can tell, Logic Apps are the cloud manifestation of BizTalk, which is an excellent product with a steep learning curve. Logic apps remove the learning curve and allow you to quickly connect and flow data through multiple systems. The session on logic apps can be seen here:

SharePoint 2016

In the past, SharePoint announcements would warrant their own post, but now SharePoint is probably best seen as part of a greater whole. Details on SharePoint 2016 details were first announced at Ignite, and I feel that the most informative session was Bill Baer’s on Wednesday morning where he outlined the major architectural changes:

Not surprisingly, this release will be very much about hybrid Sharepoint/Office 365 scenarios. Some of the notable items from the talk are:

  • SharePoint Server 2016 Will require 64 bit Windows Server 2012 or Widows Server 10, and SQL Server 2014 SP1 as a minimum
  • Standalone installations are no longer supported. It will be possible to install SharePoint and SQL Server on the same machine, but full SQL Server will be required, and SQL Express will no longer be supported. This obviously raises questions about whether or not there will be a free SharePoint Foundation SKU with the next release.
  • PerformancePoint will in fact be included with SharePoint 2016. I doubt very much that there will be any investments in it at all, but it will at least be there. I’d view this as legacy support.
  • SharePoint 2016 will support SAML claims as a first class citizen. That means that it will be possible to login with Azure Active Directory credentials, and is an example of bringing cloud identities on prem. However, don’t trash that domain controller just yet, I’m sure that service accounts will still need to be NTLM – SQL Server needs it.
  • There will be a new Roles Based installation. It will be much simpler to install and maintain servers with specific roles such as web front end, search, etc. BI will be one of the roles.
  • There will be new boundaries. Content databases up to Terabyte sizes, 10 GB file size limit, list thresholds of much greater than 5000 items (although how much greater was not specified)
  • No more FIM. The user profile engine that we’ve all grown to….. deal with from SharePoint 2010 and 2013 is no longer embedded. The full Forefront Information Manager can be used, but the default profile import mechanism will be the good ol’ User import from SharePoint 2007.
  • Durable resource based links. Every object in SharePoint will receive its own resource based URL. That means that it can be moved around in the farm, and reference URLs will still work. This is like permalinks in WordPress.
  • While not final, a preview was shown of some operational reporting. This is primarily “speeds and feeds” type information that would interest a farm administrator, although simple usage reporting could be seen.
  • Integration with the Office Graph – see below section on Delve.

SQL Server 2016

The next release of SQL Server was announces at Ignite. Its chock full of new things, focused primarily at hybrid operation and analytics. One of the more interesting concepts in this version is the ability to “stretch” a database into the cloud. With this, you can take an on-premises database, and extend it into Azure SQL, specifying rules to determine which data goes where. Given that online storage is significantly cheaper than on-premises, this makes total sense, and they’ve figured out a way to make it work reliably. The overall SQL Server keynote can be found here:

I’m very interested in the analytics capabilities, and the session outlining the improvements to SQL Server BI is found here:

I found the following items particularly notable:

  • A comment was made during the BI session that Microsoft is “Super Committed” to SQL Server Reporting Services (SSRS). Hopefully this helps quell the naysayers. SSRS is receiving a major facelift in this version, bringing a modern design experience. In addition, the parameters pane has received a great deal of attention, adding, among other things, support for cascaded dropdowns.
  • Datazen is a visualization company based in Toronto that was recently acquired by Microsoft. There is a good demo of Datazen in the session, and I highly recommend watching it. It will be included with SQL Server 2016.
  • Datazen has KPIs. It also has “sub-KPIs”. I’m not sure about you, but that sounds a lot like a scorecard to me. This may sound the eventual (see the SharePoint section) death knell for PerformancePoint, given that that’s about all that it uniquely provides to the BI stack.
  • Tabular models in SSAS (and presumably PowerPivot) will support many-many relationships and a host of other new features.
  • Tabular models in SSAS and PowerPivot will have time intelligence built in. No longer will separate time intelligence tables be required. It’s an open question however as to how extensible they will be and when.
  • SharePoint will allow browser editing on PowerPivot embedded workbooks. Currently, you need to launch Excel to edit a PowerPivot embedded workbook.

Office 365 Groups

I attended the roadmap on Office 365 Groups:

(video unavailable as of posting – should be shortly)

During this session, the light really went on for me. Groups was (were? Not sure about the grammar on this…it’s a name) introduced last year and appeared to be a glorified distribution list with Sharepoint artifacts. However, its about to become the center of the Office 365 collaborative experience. It ties together Azure Active Directory objects, a SharePoint site collection, One Note, Skype, and OneDrive into a single cohesive, non-customizable experience. It currently uses Exchange exclusively for social conversations, but full Yammer integration is promised. No date was given for the integration, but my guess is that the target is early 2016.

The current User interface is limited – too limited for my own use at the moment, but during the demonstration, a rather useful interface was shown that is coming soon. You can access groups presently through the Outlook web client in Office 365. I’m running Office 2016 preview on my laptop, and there is a very nice interface contained there. There was chatter, particularly in the Yammer community about confusion as to what tool should be used when, but I think that the coming deep integration of Yammer into Groups will render this point moot’

The next UI, demonstrated in the above session looks really good, and offers a lot of benefits. There is also a mobile app coming very shortly for, you guessed it, iOS and Windows Universal, then Android.

One unanswered question from the show is whether Groups would be available on-premises.

Power BI

Power BI content was sort of sprinkled throughout the conference, without specific focus. There was a session on the new DAX features available in Power BI Designer that is worth a watch from a modeling perspective:

One talk that really impressed me was by Lukasz Pawlowski and Josh Caplan entitled Power BI for Developers:

They cover content packs are mentioned, real time analytics, and an in depth analysis of the “how old” app that went viral during Build.

It was also announced in the SQL BI session that SSRS will in fact be included in Power BI shortly, although little detail was provided. Finally, for development, the best place to get started is http://dev.powerbi.com.

It should also be noted that Power BI was at the center of almost any analytics discussion during the conference. This is by no means a little side project.

Delve/Office Graph

Delve is a newer product in Office 365 that provides insights around what content is relevant in an organization, and how people interact with it. It’s available directly from the app launcher in Office 365, and recently, user profiles have moved to the Delve application. It’s powered by the Office Graph, which in essence an advanced index that contains content from Exchange and SharePoint, and will very shortly, be extensible for multiple content types. The roadmap session for Delve/Graph is available here:

During the session, it was stated that “Delve is the evolution of Enterprise Search”. Given that all of the work on Delve and the Graph is coming from Oslo and the former team from FAST search, this just makes sense. One of the major announcements around SharePoint 2016 was that SharePoint 2016 content can be crawled by the Office Graph to provide both search results and Delve results in Office Graph. The reverse will also be true in that the on-premises crawler will be able to index Office 365 content for search results, but Delve and the Graph will remain in Office 365. The surprise here was that later this year, it will be possible to do the same thing with Sharepoint 2013 through a coming enhancement.

Much of this Graph goodness can also now be accessed through the new Office 365 Universal API:

tyGraph

tyGraph is our product that provides advanced analytics for Yammer. It had something of a coming out party at Ignite, and while we didn’t have a booth or any launch sessions, we were fortunate enough to have several folks, customers and thought leaders present talks that at least in part featured tyGraph. If you’re interested in analytics for your Yammer network, I recommend that you watch some or all of these sessions:

Enterprise Social, from “Ooh, Shiny” to Business Success – Melanie Hohertz, Cargill

The Microsoft Enterprise Social Journey: How We Did It – Chris Slemp, Microsoft

Gain Organizational Insights with Yammer Data Mining and Analytics – Steve Nguyen, Microsoft and Tammy Young Heck, EY

Yammer Mining: Dig in and “Listen” to What Your Big *Social* Data Is Saying – Richard diZerega, Microsoft

4 Comments