Monthly Archives: April 2018

Synchronizing access to Dynamics data from external applications

What if you had to synchronize access to Dynamics data from the external applications/components. Maybe there is a scheduled process that’s scheduled to run nightly on different servers, and that’s supposed to process that data, or maybe there is a multithreaded application that’s using multiple threads.. Whatever the case is, imagine that you need to ensure that those different processes/threads do not start processing the same data in parallel.

You might think of some solution that involves external synchronization, but, to make things simple, you could just rely on Dynamics itself for this.

When registering a plugin, we can choose a stage, and, depending on the stage, the plugin might run in a database transaction. If we choose a stage where the plugin runs in a transaction, we can use such a plugin to update some kind of a well-known record in Dynamics. Since it’ll be happening in transaction, once the plugin starts, it will lock the record. So the second instance of the same plugin, which will be running in another transaction, will be locked out until the first transaction has finished.

That’s the theory, but plugins are more for server-side processing, and I started this post by describing an external application scenario.

This is where we can use a custom action:

3

For example, imagine an entity that’s maintaining a queue entity id-s that your external app needs to process. You may have many instances of that app running on different servers, for instance, and, by using this approach, you can ensure that the same queued item is not processed twice. Most of the processing will still be happening in parallel, and the only time it will be sequential is when those applications will be querying the custom action to retrieve next data id.

Now, what is that Well-Known record on the diagram? It does not matter really – it can be any record that’s always there. You might create a custom entity for that if you wanted to. Or you might use a dedicated account record..

Basically, all you need to do at the start of the plugin is this:

var entity = new Entity(“account”);
entity.Id = Guid.Parse(“…”);//Use that record’s id
service.Update(entity);

Or you might use a query to find that account.. Or you might use a dedicated custom entity. As soon as your plugin is registered in the PostOperation, this will be happening in transaction. So no two plugins will be able to do this at the same time – they’ll be doing it sequentially.

Then you can use another query to find data id and pass it back to the custom action output parameters through plugin context:

context.OutputParameters[“dataid”] = …

Last but not least, you may want to mark the records with that dataid is “processed” somehow (using statusreason.. or a dedicated field.. or somehow else), just so you the next call to the same custom action does not produce the same dataid as a result.

Then, register the plugin on your custom action message, in the post operation, and that’s about it.

As for calling the custom action from C#, here is a basic exampe:

OrganizationRequest request = new OrganizationRequest(“actionname”);
request.Parameters.Add(“Param1”, value1);

request.Parameters.Add(“ParamN”, valueN);
OrganizationResponse response = service.Execute(request);
var op1 = (string)response.Results[“dataid”];//string output parameter


Let’s face it.. sometimes we don’t care.. But how do we explain this to Dynamics?

Imagine that some data is being pushed to Dynamics through SSIS or other tool, and that data includes associations between non-existing users and their roles/teams. Your data migration script will diligently fail when trying to push those associations to Dynamics, just as it should. But what if, this time, you just wanted to ignore those errors without having to update the data migration script?

I admit this is a weird scenario, and I also admit that it goes against the rules. But, when you have little time to do something and not a lot of options, this is the kind of thing you may come up with.

Turns out that, at least in 8.2, we can register a plugin on the Associate message, we can use that plugin to remove entity references for the entities being associated from the plugin context, and Dynamics will just play along by completely ignoring that associate request.

In my case, I registered that plugin in the Pre-Validation stage assuming that it may be too late to do it in Pre-Operation. That assumption might or might not be correct, but those are finer details.. They did not matter as long as it worked.

And below is the code, just in case you ever need to do something similar. This code will check if it’s running on the “Associate” message, and, if that’s the case, it will check if either the target or the related entity is, actually, a “systemuser” entity (you will also see systemuserid there.. surprisingly, that’s how logical name shows up when a user is being added to a team). Once that’s confirmed, it will check if the user exists. And, if there is no such users, it will remove Target from the context, and it will cleanup related entities.

 

public  bool UserExists(IOrganizationService service, EntityReference userRef)
        {
            try
            {
                var user = service.Retrieve("systemuser", userRef.Id, new ColumnSet());
                return true;
            }
            catch
            {
                return false;
            }
        }

        public void Execute(IServiceProvider serviceProvider)
        {
            IPluginExecutionContext context = (IPluginExecutionContext)
                serviceProvider.GetService(typeof(IPluginExecutionContext));
            IOrganizationServiceFactory serviceFactory = (IOrganizationServiceFactory)serviceProvider.GetService(typeof(IOrganizationServiceFactory));
            IOrganizationService service = serviceFactory.CreateOrganizationService(context.UserId);
           
           if (context.MessageName == "Associate")
           {
                string relationshipName = null;
                if (context.InputParameters.Contains("Relationship"))
                {
                    relationshipName = ((Relationship)context.InputParameters["Relationship"]).SchemaName;
                }
                var targetEntity = (EntityReference)context.InputParameters["Target"];
                var relatedEntities = context.InputParameters["RelatedEntities"] as EntityReferenceCollection;
                var relatedEntity = relatedEntities[0];

               if ((targetEntity.LogicalName.ToLower() == "systemuser" || targetEntity.LogicalName.ToLower() == "systemuserid") && !UserExists(service, targetEntity))
               {
                    context.InputParameters.Remove("Target");
                    relatedEntities.Clear();
                }
                if ((relatedEntity.LogicalName.ToLower() == "systemuser" || relatedEntity.LogicalName.ToLower() == "systemuserid") && !UserExists(service, relatedEntity))
                {
                    context.InputParameters.Remove("Target");
                    relatedEntities.Clear();
                }
        }
  }

Web-based plugin registration tool, technical lessons learned

This is a continuation of the  Creating a web version of the plugin registration tool post

Let’s start with the most important part(or so I thought when I started to look into this).. How do you get a file uploaded to Dynamics without resorting to the cumbersome “use notes” technique?

There is a 3-year old post on this subject that I found very helpful:

https://css-tricks.com/drag-and-drop-file-uploading/

Basically, it describes HTML5 approach, and that’s what what I used for the proof of concept.

In the source code, you will find a JQuery dialog that accepts a file, there is some supporting code (which is more or less a copy-paste of the code demonstrated at the link above), and, then, there is a method that processes the file in the devtools_ui.js:

Sdk.UI.handleFiles = function (e) {
for (var i = 0; i < e.files.length; i++) {
var file = e.files[i];
var img = document.createElement(“img”);
img.classList.add(“obj”);
img.file = file;
var reader = new FileReader();
reader.onload = (function (aImg) {
return function (e) {
debugger;
Sdk.UI.fileData = e.target.result;
};
})(file);
reader.readAsDataURL(file);
}
}

In other words, once a file has been selected (or dropped onto the dialog), it is, then, loaded into the browser and the content of that file is assigned into the Sdk.UI.fileData property. As you can see on the screenshot below, that content is a base64 string prefixed with the data type:

image

Actually, base64 string is exactly what is supposed to be stored in the Content attribute of the PluginAssembly entity, and, eventually, it is exactly what is happening there.

BUT. This is where the limitation of web-based approach starts showing up.

There are a few entities involved into the plugin registration – have a look at the link below if you want to get more details about those:

https://msdn.microsoft.com/en-us/library/gg327902.aspx

  • Basically, we are talking about PluginAssembly and PluginType entities. This is where plugin metadata is stored, and this is exactly what your familiar plugin registration tool is using to display assembly names and related plugin types:image
  • Which means that for every assembly being uploaded the tool needs to extract plugin / workflow classes from the assembly and create required PluginType records.
  • This would not be difficult if it were C# since there are reflection classes for that, but, so far, we were only talking about JavaScript. As far as I know, using JavaScript for this task would not be possible, though, so we need to engage .NET.

 

Which means there should be a plugin, probably. Now, if there is a plugin, how do we pass that file to the plugin?

This is where nothing can really beat custom actions. In the ancient times, when custom actions were not there, we would probably create a custom entity, and we would use JavaScript to create a record.. then we would have a plugin that would be registered on the create message and that would do something. Although, once it’s all done, we would have to use JavaScript again to load that data back.

We have custom actions now, so we can actually do it differently:

  • We can define a custom action with an input parameter and an output parameter
  • We can use javascript to pass JSON string through the input parameter
  • We can use javascript to read JSON string from the output parameter (if required)
  • And we just need to have a plugin for that custom action to do whatever needs to be done on the server

 

You will find Sdk.fileUpload method in the webapi.js source file:

Sdk.uploadFile = function (fileData, onSuccess, onError) {
var uri = “/ita_fileupload”;
var resourceDetails = {};
resourceDetails.command = “uploadfile”;
resourceDetails.parameters = JSON.stringify({ fileData: fileData, assemblyId: null });
var result = Sdk.request(“POST”, uri, resourceDetails) // Adding sample data so we can query against it.
.then(function (request) {
onSuccess();
})
.catch(function (error) {
onError(error.message);
});
}

As you can see, once the “framework” is there(which, in this case, is Sdk.request method), it is super-easy to call a custom action and to pass the parameters. Actually, you will find all that “framework” code here:

https://docs.microsoft.com/en-us/dynamics365/customer-engagement/developer/webapi/web-api-functions-actions-sample-client-side-javascript

I may have done a few changes here and there when adopting it for this project, but, basically, it’s exactly that code.

Note: it does not work in IE 11 just like that. Because of what is called “javascript promises” .

I am hoping I have not lost you yet, because this was exactly the point where I hit a bit of a roadblock.

See, we can upload a file, we can pass it to a plugin, but it’s going to be a sandboxed plugin. And, even though we can use reflection there, it’s somewhat limited. Have you ever noticed that you can use the latest version of the plugin registration tool (v9) to register plugins in the 8.2 version of Dynamics? The interesting part is that, if you register a plugin compiled using V9 references, it will register ok even though it won’t work. This is because plugin registration tool will be able to parse the classes etc, it will create PluginAssembly and PluginType records in Dynamics, but, once Dynamics attempts to load that assembly, it will fail, since there will be no required references.

That’s kind of an edge case, but this is where I was not, really, sure, if I was missing some other edge cases, so I figured I’d rather try to prototype something that provides the same functionality for now. Which meant I could not just use a plugin to parse those assemblies, since, in the scenario described above, it would not work.

This is why, eventually, I ended up with this kind of approach:

image

All reflection work is offloaded to the ItAintBoring.WebApp web application (that’s one of the projects in the solution). Which, for now, is hosted in Azure.

Either way, this is just about as far as I was able to get it this time aroundSmile You can find all the sources on GitHub:

https://github.com/ashlega/ItAintBoring.DevTools

One last note on the technical side: for the web App/Service, the plugin needs to know where the service hosted, so there is a custom action in the solution that has an output parameter, and there is a workflow step that sets that output parameter. The plugin, then, calls that custom action to get the location of the webservice.

image

It’s all good except that this solution might not work too well for the upgrades. Imagine that somebody decides to deploy the solution and use a different location for the web app component. They would likely modify that custom action so it returns a different location. But, when an updated solution is deployed, it will reset that location back to the original one. In that sense, I guess using a configuration entity might be a better option.

Creating a web version of the plugin registration tool

For the last month or so I have been trying to create a web-only version of the plugin registration tool. As much as I’d like to say it’s done, it’s not the case. Looks like, at least for now, I just have to put it on hold since this has been taking an absolutely insane amount of time.

Still, there is a bunch of things I have learned/experienced first-hand, so that is what I wanted to share. What I learned in the last month.

I’ll probably split this into two different posts, though.

Technical

  • There is nothing preventing us from creating our own file upload UI elements in Dynamics. That’s right, the concept of using  “notes” as the only option for file uploads is officially deprecated(and has been for a while). HTML5 combined with javascript and plugins can do wonders
  • As of now, it seems nothing can beat custom actions when you need to create an interface between your javascripts and plugins
  • Technically, plugin registration tool seems to be using unsupported techniques to deploy plugins in Dynamics. Which is fine, though, since it’s provided as part of the SDK. Note: prior to writing this post I though I saw a reference somewhere on msdn where content attribute of the PluginAssembly entity was marked “for internal use” or something along those lines. I was not able to find that reference later, though, so it might be just my imagination
  • There are some drawbacks associated with using custom actions to store configuration settings

 

Project-related

  • When planning a Dynamics project, you have to think of the people resources carefully. The complexity of those customizations can vary a lot, and, quite frankly, this time it was a bit too much for me since I probably had to spend more time catching up on the usage of Vue, for example, than doing actual development
  • You also have to think of the architecture and consider the limitations. For example, when I’m thinking of XrmToolBox, I realize it’s a very useful tool(or should I say framework?) Still, personally I favor those tools which are deployed directly in Dynamics.. those I can use without having to install anything on my machine. Problem is, not everything is easily doable in that architecture, and, also, that kind of development requires different skillset

 

I will cover all of the above in more details, but, if you are interested in what I’d call the “proof of concept” of the web-based plugin registration tool, you will find all the sources here:

https://github.com/ashlega/ItAintBoring.DevTools

It is just that at this point – proof of concept:

image

What’s important is that it runs as a web resource in Dynamics, and you can actually upload a file and register it as a plugin assembly (including all the plugins). Whether this proof of concept will ever turn into a working “product” is a good question.. but, with that, let’s continue to the technical lessons learned