Monthly Archives: August 2018

Dynamics Portals: The most useful caching tip I found so far

I’ve been struggling with portal caching lately – I would update some data in Dynamics, and that change would not be reflected on the portal right away. Unfortunately, there are times when this behavior is just not acceptable since it may lead to all kind of negative feedback from the portal users.

And, then, it seems that an ultimate solution presented itself in the form of this tip of the day:

https://crmtipoftheday.com/784/some-finesse-in-portal-cache-resets/

I am not talking about publishing all customizations or about invalidating the cache.. I’m talking about the Change Tracking option:

image

Enable it on the entity, publish all.. Refresh the cache as described here:

https://docs.microsoft.com/en-us/dynamics365/customer-engagement/portals/clear-server-side-cache

Maybe refresh it twice. And, then, enjoy the results. To be fair, it seems that, occasionally, some data may still be cached “incorrectly”, at least that’s the impression I got so far. Still, in 9 out of 10 cases it seems to be working once that change is in place.

I’d really love this tip to be mentioned in the portal documentation..

Dynamics Portals: Liquid Reference (the missing Appendix A)

 

If you are starting to work with Liquid, Microsoft has a rather detailed documentation here:

https://docs.microsoft.com/en-us/dynamics365/customer-engagement/portals/custom-templates-dynamic-content

But, after a few days of having to scan through that documentation, I’ve started to realize that it would be great to also have a reference page that would list all Liquid keywords and cross-reference them with the corresponding documentation pages. The purpose of such page would be to show me, quickly, what’s available. In one place. So that I would not have to scan through the documentation just to figure out if there is something I could use.

If it were a technical book, this kind of reference would probably go to the Appendix A, and, hence, that’s what is is.. the missing Appendix A (and you know what? I did not realize this thing is going to be that long, yet there is a part about liquid objects that’s still not finished):

LIQUID OPERATORS (Click here for the details)

==      Equals

!=
       Does not equal

>
        Greater than

<
         Less than

>=
      Greater than or equal to

<=
      Less than or equal to

Or
      Condition A or condition B

And
    Condition A and condition B

startswith  tests whether a string starts with a given substring.

endswith    tests whether a string ends with a given substring

contains      tests for the presence of a substring within a string

 

LIQUID TYPES(Click here for the details)

String, Number, Boolean, Array, Dictionary, DateTime, Null

 

LIQUID CONTROL FLOW(Click here for the details)

if,else,elseif,endif; unless,endunless; case,when,endcase

 

LIQUID ITERATION TAGS (Click here for the details)

for/endfor; cycle; tablerow

 

VARIABLE TAGS (Click here for the details)

assign          Creates a new variable

capture       Captures the content within its block and assigns it to a variable

 

TEMPLATE TAGS (Click here for the details)

include                  Includes the contents of one template in another, by name

block/extends   Provides template inheritance

comment             Allows you to leave un-rendered code inside a Liquid template

 

ENTITY TAGS (Click here for the details)

chart                      Adds a Dynamics 365 chart to a web page

editable
                Renders a given Dynamics 365 portals CMS object as editable

entityview
            Loads a given Dynamics 365 view

searchindex
         Performs a query against the portal search index

entityform
           Fully renders a Dynamics 365-configured entity form

 

LIQUID FILTERS (Click here for the details)

batch                        Divides an array into multiple arrays of a given size

concat
                      Concatenates two arrays into a single new array

except
                      Select all the objects in an array where a given attribute does not have a given value

first
                           Returns the first element of an array

group_by
                Group the items in an array by a given attribute

join
                            Joins the elements of an array with the character passed as the parameter

last
                            Returns the last element of an array

order_by
                 Returns the elements of an array ordered by a given attribute of the elements of the array

random
                    Returns a single randomly-selected item from the array

select
                        Selects the value of a given attribute for each item in an array, and returns these values as an array

shuffle
                      Applied to an array, returns a new array with the same items, in randomized order

size
                             Returns the number of items in an array

skip
                             Skips a given number of items in an array, and returns the rest

take
                             Takes a given number of items from the array, returning the taken items

then_by
                     Adds additional subsequent ordering to an array already ordered byorder_by

where
                         Select all the objects in an array where a given attribute has a given value

date
                             Formats a DateTime value using a .NET format string

date_add_days
        Adds the specified number of whole and fractional days to the DateTime value

date_add_hours
      Adds the specified number of whole and fractional hours to the DateTime value

date_add_minutes
Adds the specified number of whole and fractional minutes to the DateTime value

date_add_months
  Adds the specified number of whole months to the DateTime value

date_add_seconds
Adds the specified number of whole and fractional seconds to the DateTime value

date_add_years
       Adds the specified number of whole years to the DateTime value

date_to_iso8601
     Formats a DateTime value according to the ISO 8601 standard

date_to_rfc822
        Formats a DateTime value according to the RFC 822 standard

current_sort
             Given a sort expression, returns the current sort direction for a given attribute

metafilters
                Parses an entitylist filter_definition JSON value into filter option group objects

reverse_sort
             Given a sort direction, returns the opposite sort direction

ceil
                                Rounds a value up to the nearest integer

divided_by
                 Divides a number by another number

floor
                             Rounds a value down to the nearest integer

minus
                           Subtracts a number from another number

modulo
                       Divides a number by another number and returns the remainder

plus
                              Adds a number to another number

round
                          Rounds a value to the nearest integer or specified number of decimals

times
                           Multiplies a number by another number

append
                       Appends a string to the end of another string

capitalize
                   Capitalizes the first word in a string

downcase
                  Converts a string into lowercase

escape
                        HTML-escapes a string

newline_to_br
         Inserts a <br /> line break HTML tag at each line break in a string

prepend
                     Prepends a string to the beginning of another string

remove
                       Remove all occurrences of a substring from a string

remove_first
            Removes the first occurrence of a substring from a string

replace
                       Replaces all occurrences of a string with a substring

replace_first
            Replaces the first occurrence of a string with a substring

split
                             The split filter takes on a substring as a parameter

strip_html
                 Strips all HTML tags from a string

strip_newlines
        Strips any line breaks from a string

text_to_html
           Formats a plain text string as simple HTML

truncate
                    Truncates a string down to a given number of characters

truncate_words
     Truncates a string down to a given number of words

upcase
                       Converts a string into uppercase

url_escape
               URI-escape a string, for inclusion in a URL

xml_escape
             XML-escape a string, for inclusion in XML output

boolean
                    Attempts to convert a string value into a Boolean

decimal
                     Attempts to convert a string value into a decimal number

integer
                      Attempts to convert a string value into an integer

string
                         Attempts to convert a value into its string representation

add_query
               Appends a query string parameter to a URL

base
                           Gets the base URL of a given URL

host
                            Gets the host part of a URL

path
                           Gets the path part of a URL

path_and_query
   Gets the path and query part of a URL

port
                           Gets the port number of a URL

remove_query
       Removes a query string parameter from a URL

scheme
                     Gets the scheme part of a URL

default
                      Returns a default value for any variable with no assigned value (i.e. null)

file_size
                    Applied to a number value representing a number of bytes, returns a formatted file size with a unit of appropriate scale

has_role
                   Applied to a user, returns true if the user belongs to the given role. Returns false if not

 

LIQUID OBJECTS (Click here for the details)

THAT’s a long list here.. coming soon

 

LIQUID.. UNDOCUMENTED (Except by Colin Vermander. Click here for the details)

fetchxml                 Query data from Dynamics using fetch xml

Dynamics Portals: Adding a custom button to the entity forms

Was looking into how to add a custom button to the entity form, and found this thread in the community forums:

https://community.dynamics.com/crm/f/117/t/274955

So with a little modification(class attribute of the new button element), just so the second button looks similar to the default “submit” (which is re-labelled to “Subscribe” on the screenshot below), here is the script:

$(document).ready(function(){

$(‘#UpdateButton’).after(‘<button id=”deletebutton” type=”button” class=”btn btn-primary button” style=”margin-left:10px;”><span></span> Unsubscribe</button>’);

$(“#deletebutton”).click(function(){
    $(‘#ita_subscribe_0’).attr(‘checked’, ‘checked’);
    $(‘#UpdateButton’).click();
});

}

image

Then you can do whatever you need in that onclick javascript (set field values etc), and, eventually, just pass onclick to the default UpdateButton.

Dynamics Portals: Basic relationships

As I mentioned to somebody else the other day, I’m relatively new to the portals. So this blog post might not be extremely revealing for somebody who used to be working with the portals for a while. On the other hand, while Microsoft does have a very extensive documentation on the portals, personally I’d love to see a bit of a high level overview first so I could put all other information into the appropriate buckets. I used to work on the custom portal implementations, so I can relate to some of the techniques / issues / solutions implemented in the former ADX / current Dynamics portals, but what I’m sort of missing is the introduction into what fits where in the data model.

So, with that said, let’s see if I can pull off my own introductory post.. And forgive me Adoxio people, I feel like I’m trespassing nowSmile

Anyway, when looking at the portal entities in Dynamics, a lot of them just make sense:

image

Intuitively, we know what a redirect is, what site settings are, what invitations are.. the same goes for ads, polls, poll placements, etc. You can look at many of them, read on quickly, and you’ll get a pretty good idea of what they are for.

Where it may get confusing, at least it did for me, is with these few:

  • Web Sites
  • Web Pages
  • Page Templates
  • Web Templates
  • Web Forms
  • Entity Forms
  • Entity Lists
  • Web Files

 

What’s a web page and how is it different from page template? How is page template different from web template? What’s a web form and how is it different from entity form?

Especially since most of those show up on this single screen:

image

So, before I continue.. What about the Web Files and Web Templates? They don’t show up on the screenshot.

With the web files, it’s relatively simple: as per the documentation, “A web file represents downloadable file in a portals website, used to store images, documents, and any other file type”. This includes CSS, png, jpg, etc files. In Dynamics terms, those would be web resources.

Web Templates are more complicated – they are part of the mix above, so we’ll get to them down below.

Now let’s start defining the purpose of those entities from “simple” to “complex” (technically, they should probably be arranged differently, but I’m only talking about the conceptual meaning).

Web Site

This is all about web site metadata including the header and footer references, site settings, access permissions, roles, linked page templates and content snippets, etc. Normally, there is 1 web site per environment (unless, I guess, you wanted to be able to switch between sites). Look at it this way.. Your portal deployment is just generic code – in order to display an actual web site it needs site-specific details. And you provide those details to the portal by provisioning a WebSite record in Dynamics.

Now, if a web site is more like an overarching entity, the 3 entities that follow are sort of page “content controls” since we can put them on the web pages:

1. Entity Form

One of the main benefits of using the portals is that we can design a form in Dynamics and, then, expose it on the portal. However, we can have a lot of entity forms in Dynamics, and we need to tell the portal, somehow, which one we want to use on the portal. This is what we can do using the Entity Forms. And, while doing it, we can also define some portal-specific behavior such as whether we want to generate steps from tabs, what we want to do on success, whether we want to add some javascript when the form is displayed, etc. But, essentially, it’s a “content control” for the portal, since you can use the same entity form on different pages.image

2. Entity List

Just like with the entity forms, we can quickly expose a list of records, or a grid. And, just like with the entity forms, entity lists are built on top of Dynamics native metadata – for each entity list we have to identify the entity and the views. Other than that, there are some portal-specific settings such as which buttons to show, whether to display search, whether to add custom javascript, etc. Still, it’s yet another content control.

3. Web Form

Where an Entity Form represents a single form, a Web Form represents multiple Dynamics forms (or tabs from those forms). Each of those individual forms/tabs is represented by a Web Form Step. In other words, it’s, basically, a step-by-step experience.

A Web Form is a holder for all those individual steps, and every individual step is somewhat similar to an Entity Form.. but we are getting some multi-step specific settings and perks (“next step”, related web form sessions, etc)

By this moment what’s left of the original list is: Web Pages, Web Templates, Page Templates. So we are doing well – it’s down to 3 entities.

Now, I mentioned “content control” a few times before. This is not an official term, but I’d like to keep using for simplicity because here is how all those entities could be categorized, then:

  • Content Controls: Entity Form, Entity List, Web Form
  • Metadata: Web Site, Web Page, Page Template
  • HTML: Web Template

Let’s have a look at the HTML category first

Web Template

This is your actual “HTML”, or the template. Conceptually, this can’t be any simpler – any web page should be displaying some HTML, whether it’s static or dynamic:

image

Of course that source code on the screenshot is far from being simple, but we are just trying to understand the purpose of the entities. Not the details of how to work with them.

Finally, let’s get back to the Metadata. We’ve looked at the Web Site, but, somehow, I have two more entities there: Page Template and Web Page. And, I think, this is where I may still be getting it somewhat wrong.

Every Page Template may have multiple associated Web Pages. Every Web Page has to have a Page Template. A Web Page is more about the portal structure, properties, and associated content controls – this is where you can define child pages, url, publishing state, associated web form/entity form/entity list, exclude that page from search, hide it from the sitemap, etc.

Whereas a Page Template is more about HTML metadata: this is where you can define the title, specify if the header/footer will be displayed, etc.

Given that a page template has only a few attributes (see below), what is the reason there is a special entity for that – why could not all those attributed be placed directly on the Web Page entity? Other than providing  an additional level of abstraction/decoupling, I can’t think of any other reason right away.. This is where a good example would be handy, but, since I don’t have one, let’s just say that Web Page is what you need to create when defining a new portal page, and, in order to link that new web page to an actual Web Template, you need to use a Page Template (besides, this is where you can also say if you want to use header/footer for that web page or not).

image

image

Hope that helps.. Have fun with the portals!

Portal Content Editor: Preview On/Off and caching

When working with the portal content editor, we have the option of turning the preview on and off:

image

That setting is supposed to either display “draft” content (preview ON) or to hide it(preview OFF):

https://docs.microsoft.com/en-us/dynamics365/customer-engagement/portals/get-started-portal-content-editor

That’s what I thought, but it took me a few hours (and, eventually, some timely guidance) to realize that Preview ON/OFF selection is, actually, cached. Or, at least, some of that is cached somehow.

See, on the screenshot above “Forums” link is not supposed to be showing up – it’s in the draft publishing state:

image

But I can make it disappear by refreshing the cache:

https://docs.microsoft.com/en-us/dynamics365/customer-engagement/portals/clear-server-side-cache

Once it’s been done, there is no link:

image

Of course, once I’ve done that, I can’t just turn the preview ON to see the link:

image

Have to refresh the cache again(or I might probably choose to wait for 15+ minutes till the cache is refreshed automatically):

image

Field Security profiles, unmanaged solutions, and why we should actually be looking at the warnings in the solution import logs

There is an interesting problem that we’ve just discovered. Something fishy has been going on with the field security profiles in our solutions – some time ago we’ve noticed that, every now and then, some of the field security profiles would not be properly updated in the destination environment after importing our updated solution there.

At first we thought maybe it was just a glitch, but, a few weeks later, it happened again. So it was a somehow persistent glitch which was becoming a bit of a problem.

A colleague of mine suggested that there might be something in the solution import logs. Those logs are always filled up with some warnings about workflows and sdk message processing steps, and it’s a big solution.. so more often than not we just don’t pay attention. It’s an unmanaged solution, btw.

Well, once the problem happened again.. Actually, one we’ve discovered the problem again (there is a difference), we’ve tried re-importing the same solution to the environment where the problem just happened. There was nothing about the security profiles in the logs. No luck? Was it really a glitch happening whenever it wanted to happen?

Luckily, we had a backup of the database taken before our updated solution was deployed, and, since we are on-prem, we tried restoring the database and importing the solution there.

There was our glitch, right in the logs:

image

Now this was all happening in 8.2 (and this is also important).

After a bit more testing, here is what we’ve discovered:

  • You can add a new field to the solution, enable field security on that field, and add it to a field security profile. Then you can export that solution and import it to the destination. It will work.
  • However, if you already have a field in both environments, but field security is not, yet, enabled in the target environment, importing the solution will lead to a warning above.

This is because a field can be added to field security profile only after that field has been enabled for field security, and, if it’s an existing field, that kind of change has to be published first. This is different from when a field is being created with field security enabled – that operation is automatically published.

And you know what else is interesting? This is not a problem.. well, sort of.. anymore in v9, since, if you run into this situation in V9, you’ll get an error (not a warning), so solution import will simply fail:

image

Which is probably a better option since, at least, you’ll know that it did not work – won’t have to wait till your users discover the problem.

Still, whether it’s V8 or V9, we may need some kind of a workaround, and, it seems, the easiest one would be to move Field Security Profiles into a separate solution, so the whole import process would look like this:

  • Import the main solution (which would not include field security profiles)
  • Publish all customizations (field security will be enabled on the secured fields after this)
  • Import the second solution which would contain your field security profiles

PS. And what about the managed solutions? I suspect they should work just fine in the same scenario since changes imported through the managed solutions are published automatically. So a field would become enabled for field security profile, and, then, Dynamics would be able to configure the profile(s) without issues.

How to: delete entity attributes using Excel import in Dynamics

Note: this solution was updated to v 1.0.1 on Aug 9. Instead of the original workflow activity, there is a plugin and an SDK step now. It turned out custom workflow activity was causing Generic SQL Errors in some environments.

If you just asked “what?!” after reading the title, I guess my mission has already been half accomplished. Catchy subjects and all that..

But, seriously, we don’t always have the luxury of automated deployment on our projects, so we often have to do a lot of stuff manually when maintaining Dynamics environments. And one of those things which I found causing a bit of a problem is making sure that all the attributes that were supposed to be deleted have, actually, been deleted.

As in.. While working in the sandbox environment, we may choose to delete an attribute. But we also need to remember to delete it from production when the time comes. And, of course, in the absence of automated deployment, deleting an attribute is, often, the last of our concerns since, after all, having an extra attribute is unlikely to break anything. And going over a bunch of attributes does take time.. not surprisingly, we may simply choose to keep them in production this time.. and the next time.. and, eventually, we won’t know what’s needed and what’s not anymore.

So, to make it a bit easier, what if we could list all deleted attributes in a spreadsheet, and, then, import that spreadsheet to the production environment? Of course it would not help much unless there were some code that would process the spreadsheet to actually delete the attributes.

This is exactly what “Solution Management” project is about:

https://github.com/ashlega/SolutionManagement

Here is how it works:

1. You will need to download and deploy the managed solution (or you can use unmanaged – totally up to you)

image

2. With this solution, a few components will be deployed

  • Solution Action entity

 

image

  • A plugin and an SDK step registered to run OnCreate (pre-validate) of the Solution Activity

3. To use the solution, you would need to create Solution Action records in the production environments, so the workflow would take care of deleting the attributes. Using excel import might be one of the easiest options for that

  • Start by downloading the import template for the Solution Action entity

 

image

  • Open the spreadsheet and fill in the data – you only need to provide Action, Entity, and Attribute names

 

image

  • This is important: make sure you have deployed the customizations which remove dependencies for the attributes being deleted. The workflow will not be deleting the dependencies – it will only try to delete the attribute. If this sounds like a limitation, think of it this way: assuming you are not using the attribute, there should be no dependencies. By this time, you would normally have worked through that while customizing the forms/views/charts/dashboards, etc. In other words, this is not something you need to do manually in production – you just need to bring over your customizations (as a solution) from sandbox/development to production.
  • Finally, import the spreadsheet to Dynamics

 

image

You may want to give the workflow a few minutes to complete, since it’s a background workflow. And, then, you can go to the advanced find and have a look at the Solution Actions:

image

Believe it or not.. We’ve just deleted an attribute using Excel import!

 

Error logging for Microsoft Dynamics Portals

 

Just got some time over the weekend to look into the error logging feature for the portals – it was introduced a while ago, so, if you have not seen the documentation yet, have a look:

https://docs.microsoft.com/en-us/dynamics365/customer-engagement/portals/view-portal-error-log

This turned out to be a mixed experience, although, it’s certainly better to have logging than not to have it.

Basically, there are two different features:

  • There is server-side error logging
  • And there is this new ability to display plugin/workflow errors right on the portal

As of this moment, here is what you may expect when using that functionality:

1. Server side error logging

One thing to keep in mind when setting up the connection is that you need to copy connection string from the blob settings in your Azure portal:

image

If you make a mistake there, portal configuration screen will be asking you for the valid URL. Which is a bit misleading – it really needs the connection string from the screen above:

image

  • BTW, just in case it matters, you can use any Azure tenant for the blob.

Once it’s all been setup, you can see error logs in the Storage Explorer right in the Azure Portal:

image

I am not sure why there are two folders, so try both.. There are subfolders organized by month/day, but, in the end, you’ll find a CSV file:

image

Download the CSV, and you’ll the errors. For some reason, there will be other errors in the logs (for the lack of better word, I’ll call them “system errors” for now.. most of them seem to be some kind of file system “access denied”), but, among those errors, you’ll see the one produced by your real-time worklfows/plugins, for example:

image

2. Client-side error notifications

This is where we should be able to create a new site setting ( Site/EnableCustomPluginError ) to see a friendly error message when an error happens in a plugin:

image

This has not worked out for me, yet, even with the site settings added/configured, so it might not be working in my version of the portals (9.0.6.1). Still, it’s probably going to be even more useful than server side logging since, out of a sudden, we are getting a simple solution for communicating server-side business logic validation errors to the portal users.

EasyRepro Tips: why would you ask your test to spend time just thinking?

 

If you ever looked at the EasyRepro samples, you’ve probably seen a lot of calls to the ThinkTime method. Here is an example – basically, you do something useful in the test.. and, then, you add ThinkTime:

image

What ThinkTime does – it’s just puts current execution thread on hold for the specified number of milliseconds:

image

Hm.. you probably did not want your test to spend useful automation time sleeping, right? Well, of course you can take a nap, too, but that definitely looks suspicious.

So, what is the purpose of this? First of all, let’s remember that we are using EasyRepro (and, therefore, Selenium) for UI testing. When an actual user goes to Dynamics to do something, that user will never be doing everything with the speed of a robot talking to the computer through a wired interface – instead, there will be mouse movement time, there will be unexpected browser slowdowns, there will be urgent phone calls that will break the flow, etc. So, if the purpose of our UI test were to emulate user actions, then it would be only reasonable to introduce some delays.

Still, that would be a good explanation, but is there a technical reason, too?

When I was looking into it, my first thought was that, possibly, it’s required to ensure that, whatever javscript event handlers are there in Dynamics, we are giving them enough time to run before we do anything else. However, right now I think it’s not the main reason. There is no multithreading in JavaScript, yet both UI and programmatic events are handled on the same main thread. So.. If we use EasyRepro to set a value of any particular field, all required event handlers will be added to the queue right away, and, if we do anything else in our test code, that action will be added to the queue right after the event handlers. Since EasyRepro is, actually, using a programmatic interface (for instance, you won’t see mouse movements while running the test.. you can actually keep doing something on a different screen/in a different application while the test is running), those events will just be added to the queue.

Actually, I just went to one of the sample tests (“CreateNewAccount”) and removed all ThinkTimes. That version of the test worked just fine, too:

image

See that message on the left which is saying “Test Passed”? Now let’s think of what was happening in that test:

  • We have emulating user login
  • We have navigated to Sales->Accounts
  • We have changed the view
  • We have clicked “New” button in the command bar
  • And we have populated a few fields

 

The interesting part is clicking that “New” button, since that’s when a completely new screen was displayed in CRM. So how did EasyRepro knew NOT to try setting a value for “name” field before that new form was loaded?

This is because EasyRepro was designed with this scenario in mind. If you look at the implementation of SetValue method, you’ll see how one of the calls there is driver.WaitUntilVisible:

image

Which will, apparently, wait until the element we are trying to set a value for is visible. And the other fail-safe embedded into the EasyRepro framework is that it’ll often wait until the page is fully loaded before returning control to your test code. For example, this is what’s happening right in the end of the ClickCommandButton code – see that call to WaitForPageToLoad?

image

So, to summarize:

  • ThinkTime may be needed if you really want to emulate user behavior. As in, if you want to slowdown the automated test (may be if you don’t want it to turn into a load test, especially if there are plugins/workflows involved. Although, it’s not that a single test session should be able to break Dynamics that way, so.. really there might be no need for this)
  • ThinkTime may be needed if you wanted to implement a quick and dirty workaround where there are some events/actions which are not covered by EasyRepro. For example, you may have your own script which is doing ajax processing and which is normally fast, but it’s not fast enough for the automated testing. Although, a better option in that case might be to figure out some other condition, such as to check element visibility, or even to run an additional javascript through the web driver to verify the completion of your ajax call
  • Finally, you may need to add some extra ThinkTime if, for whatever reason(maybe you have plugins, maybe you have workflows, etc), Dynamics is not responding fast enough. EasyRepro defines a number of default timeouts in the Constants.cs file, and, if those timeouts are not sufficient for a particular action, your tests may start failing:image