In the previous post, we saw how to create and train a Form Processing model, a tool released in preview last June and available in the AI Builder box of Power Automate.
In this second post, we are going to use that model in a Flow, so that it will take files from a specific repository, analyze and extract data from it, and store it in a final location associated with the extracted data
The source where the files will be picked up is going to be a OneDrive folder whereas, on the other end, the destination will be a SharePoint Online document library.
As such, in this post we will :
- Create a SharePoint Online document library able to store the file's metadata
- Create the Flow
- Test the end-to-end scenario
From the previous post, the following document properties will have to be stored, and thus, the corresponding columns in the document library (for simplicity, all of them are going to be a "Single Line of Text" type of property) :
- Last Name
- First Name
For the source, just create a OneDrive for Business folder.
Create a Power Automate Flow
Switch to Power Automate and create a new solution, or use an existing one, give it a name, select a publisher and set its version (all these properties are mandatory). In fact, AI Builder models can only be used from within a solution-aware Flow (reference here).
Once in the selected, or created, solution, we can add a new Flow.
As a trigger, we select the "When a file is created" from the "OneDrive for Business" category. It needs to have the permissions to read the files in the configured directory, thus, credentials may be asked to get access to the folder.
This trigger will thus launch the Flow whenever a new file arrives in this folder.
When the Flow gets the file, it is time to use the model created in the previous blog post. Let's select the "Predict" action from the "Common Data Service" group, we can see that it is indeed a Premium feature. In the configuration of the "Predict" action, we can now select the model we have built and named, in this case, "My Request Form Model". For the "Request Payload" JSON parameter, we have to take the content of the HTTP request's body, which gives the following :
The call to the model will return a JSON response that requires to be parsed. Obviously, we should use the "Parse JSON" action to extract the different value from the response. But, for this, we must specify the schema, and the best way to get it is to provide an example of the data returned by "Predict".
We can have such response by saving the Flow as it is now, and to test it. Let's click on "Test" and using an example of a form to be dropped in the OneDrive folder. After several seconds, the Flow will run, and hopefully, succeed. If we click on the execution history, we can see the different inputs and outputs for all the executed actions. Then, if we take the "Predict" action, we can copy the "Output" of it in the clipboard.
Now, we go back to the edition of our Flow, and add the "Parse JSON" action, set the "Response Payload" as a "Content" parameter.
For the "Schema", we click on the "Generate from sample" which will pop-up a window where we can paste what we have in the clipboard. After clicking the "Done" button, the "Schema" parameter is filled.
Everything should be fine would we say.
But, unfortunately not. It happens that the schema generator is misleading by the data we give to it. A typical example is the "confidence" value. If all your fields during the analysis of the model returned 100% of confidence, it means the data returned will be equal to "1". The consequence is that the schema generator will interpret it as an integer, which is wrong. It is enough that once the model returns a confidence of less than 100% to prevent any schema validation. So, it is necessary to replace all confidence fields of type integer by number, to be safe.
We are done with the "Parse JSON", now, let's create the file.
For that, we will use the "Create file" action from the "SharePoint Online" group, then we select the "Site Address", the "Folder Path" (which is nothing but the target document library), and we set the "File Name" and "File Content" parameters to the corresponding attributes of the source file.
This action can't update the properties of the file, leading us to add another action called "Update file properties" from the same "SharePoint Online" group.
As for the previous action, the "Site Address" and the "Library Name" must be set, but rather than specifying the file name, we have to use the "ItemId" from the previous action to define the file that needs to be updated. The columns added to the library are automatically present as parameters of the action, but the difficulty here is to find the right "value" to set in each of these fields. There is apparently a bug as the value selector only displays the "value" attribute's name, which is "value", instead of taking the parent attribute name that would be "LastName", "FirstName" and so on. Nevertheless, before picking randomly a value and hoping for the best, we can make a quick check once we have picked a dynamic field. By hovering the "value" field with the mouse, a tip is popping up, displaying the complete expression, containing the real attribute name (here "LastName").
Finally, once all these file attributes are set, it is time to save the Flow, and test it. Just drop an PDF form in the OneDrive for Business folder, wait for some seconds or minutes, and the file should appear in the SharePoint Online document library with the correct metadata. Or, almost.
As can be seen in the screenshot, the LastName property is not well extracted, and this comes back to my first suggestion for improvement where the field zone should be manually adjustable to resize the location where the data can be. In this case, we could see that the "LastName" zone was far too large, leading to a property that is not only containing the name, but also part of the form instructions.
But, otherwise, we can see that the document has the correct properties taken from the content of the PDF.
Last June, Microsoft introduced AI Builder, which is an artificial platform that can be integrated in a Power Platform low-code application. It is still in preview, and since that day, I wanted to give it a try and find an interesting use case to test these new features.
In the context of automation of office document and application processing, AI has perfectly its place and we can see that Microsoft is making important investments in that area.
The use case I took for this post is to automate the processing of request forms in order to extract the data from them and to use them as metadata in SharePoint. Of course, these metadata could directly be used to make decision or to go further in the processing of a request, but let's keep simple first.
Thus, the steps are :
- Get the file
- Analyze the file and extract the data
- Upload the file in SharePoint
- Update the file with the metadata
This post and the next one will then guide through the steps to create this flow
Create a Form Processing Model
The first thing to do is to create a Form Processing model that will be used later (in the next post) by the Flow.
At the time this post is written, Microsoft proposes 4 models ready to be used :
- Form Processing to read and process documents or forms
- Object Detection to analyze images
- Prediction in order to give an idea of what will happen, based on past data
- Text classification in order to analyze its meaning or even make sentiment analysis
The starter model to use in this case is of course the Form Processing one.
These models are premium features, which means that you'll probably have to activate them or start a 30-days trial period. Unfortunately, after the 30 days of trial, it seems that the models wouldn't be usable anymore
Creating a Form Processing model is done in several steps, but before starting the creation of the model, you will need a minimum of 5 forms to train it. It is expected that the forms have the same layout. Indeed, the model will try to identify the zones were the data is, therefore, these data must be in the same locations on the form. An advantage is that it is also able to identify zones where hand-written inputs are present. In other words, the OCR seems working pretty well.
If the forms are already available, when starting the wizard, the very first step of the Form Processing model creation is simply to give it a name. At this stage, there are also some explanations of the constraints for the model to work, like the one mentioned just before about the layout.
The next step asks to upload the minimum 5 forms that will be used to identify the zones where the data is. To have a better result, I would advise to use the best forms available, in terms of clarity, contrast, etc, which would help the model in the identification of the interesting zones.
After this step, there is still a way to upload other samples of the forms, but the limit is set to a total upload of 4MB. This limit, for the time being, seems a bit low, but, with 5 or 6 documents, the results are already good. According to this post, it is a feature that will be improved.
The next step is the analysis of the uploaded documents that will lead to the selection of the fields we want to get from the forms.
It shows one of the uploaded forms, with the discovered zones of data and a percentage of accuracy estimated by the model. As the example shows, almost all the fields have been correctly identified, except the first one, where the zone is a bit too large.
Here, I would make one suggestion of improvements : either we should be able, at this stage, to come back to upload additional samples or at this stage, or, to be able manually to resize the zones. This is important, because having a zone too large will make the extraction of data less accurate (we will see what it means later in the second post). Apparently, this is something that will come.
In addition to selecting the fields, we can give and change the names of these fields.
Once this is done, we are back to the list of models we have created, and what remains to be done is to make our model available for use. This step is called publishing. A click on the model will allow us to either make a quick test by uploading a form sample, or to publish it.
Here, I would also have a to be able to update the model, for example after having improved it and to be able to publish further versions of the model.
As we can see, creating a model to process forms is quite easy, thanks to the AI in the background, making these capabilities at our fingertips. It is still in preview and no doubts they will be perfectioned in a near future, but it shows that automation is something taken seriously at Microsoft. The process of creating a model and how it can be used in Flow is really easy and makes it accessible also for the users that don't have too much technical knowledge in artificial intelligence.
We will see in the next post how to use it in Flow, in order to automate the processing of PDF documents.
Is this blog still alive ?
Well, after more than 3 years of inactivity, it is a legitimate question.
In these last 3 years many things happened as one can imagine, also within the Atos organization in Switzerland (the one I am working for, in case this wasn't yet known).
As such, following the possibilities of growth in the company, I have been involved in projects and initiatives much more important in size than I used to. This made me diversify once again my focus on other technologies.
A first example is that I have recently been accepted as a member of the Atos Expert Community for the Cloud Platform domain. To summarize, the Atos Expert Community is a network of more than 2'700 experts around the world on 13 domains. They focus on innovation and experiments in their technology area. The activities of this network can be followed on Twitter by using the hashtag #YourExpert
Another important example, I got very recently certified Google Cloud Platform Professional Architect, And I will make a specific post on this to explain the ways I prepared for the exam shortly
In parallel, I also gained substantial knowledge on Amazon AWS. These activities on such different platforms completed the knowledge I already had on the Microsoft Azure one, which is not only interesting but also important when proposing Cloud solutions to the clients. Simply because the price of the cloud services is not the only driver when selecting one provider or another and knowing the benefits or challenges of the 3 main players in the Cloud is an important asset.
It was already visible in the last post, artificial intelligence and machine learning took more space in my areas on interest, therefore more content about these themes will come on the blog too.
A as consequence, there will be less SharePoint related posts (this was already the case before this 3-years "break"), and more cloud and architecture subjects.
I also intend to share more frequently little posts on what I found interesting or the experiments I made during a week, that we could name #WhatILearnedThisWeek. The goal is to not give an in-depth opinion or explanation of a topic or a technology, but rather give a quick overview of it.
On a totally non-technical perspective, I continue working on the public speaking activities and the most recent example was my participation to the Swiss Travel Management Forum where I had the pleasure to explain the importance of artificial intelligence for the business travel.
In addition, this is new, I will share some on the modern workplace topic. In this area, I intend to share more of my thoughts about remote working, ways of working (remotely or not) and the technologies that help being efficient at work.
Currently having some Linux machines on which DeployR Enterprise 8.0.5 must be deployed, I faced a nice bug in the DeployR configuration utility.
First, I started to deploy a “Microsoft R Server on Linux” from the Azure marketplace. The setup is rather simple, and quite fast, and the original setup would be fine if you want to use the default ports for DeployR and RServ.
For the little story, the default ports have changed between the version 8.0.0 and the version 8.0.5. Indeed, whereas in the version 8.0.0 the default ports were 8000, 8001 and 8004 for DeployR, DeployR with SSL and RServ respectively, Microsoft adapted the default port numbers to the new version number. Thus, the new default ports are 8050, 8051 and 8054 for the same services. Basically, they used the first 3 digits for the version.
But, if you want to change these ports, you can do it by running the utility /opt/deployr/8.0.5/deployr/tools/adminUtilities.sh .
Selecting the menu “6. Change DeployR Ports”, the utility offers the following options :
1 DeployR Port Option (Sub-Menu)
3 A. Change Tomcat Connect Port (Currently: 8050)
4 B. Change Tomcat SSL Port (Currently: 8051)
5 C. Change Tomcat Shutdown Port (Currently: 8052)
6 D. Change DeployR Rserve Connect Port (Currently: 8054)
7 E. Change DeployR Event Port (Currently: 8056)
9 M Back to Main Menu
10 Q Quit
12 Enter an option:
All the options work, EXCEPT the option “D”. If you try to configure the option “D”, it will completely overwrite the tomcat configuration file (located in /opt/deployr/8.0.5/tomcat/tomcat7/conf/server.xml .
What it does is that it replaces the original content by what should be the new configuration of the RServ service (below, I wanted to change the port number to 8004). In other words, the content of the server.xml file would be :
1 interactive off
2 source /opt/deployr/8.0.5/rserve/RScripts/source.R
3 encoding utf8
4 remote disable
5 port 8004
6 workdir /opt/deployr/8.0.5/rserve/workdir/Rserv8.0.5
So, before doing any mistake, make a copy of the tomcat configuration file, and, if you change the port number of RServ, restore the configuration file from the copy after the change in the utility. And also, make the change manually in the /opt/deployr/8.0.5/rserve/Rserv.conf
I am quite used to give public speeches, as multiple times I was speaker at conferences or internal meetings. But, several weeks ago, the speech I gave had mainly 2 objectives : giving an overview of IBIS, and, promoting our internal Toastmasters club. For this occasion, I thought it would be interesting to share how I usually prepare speeches, taking this recent meeting as an example.
To set the context, the primary objective was to promote our internal Toastmaster club, open to all our employees (but not only), during a 45 minutes presentation of what is public speaking, giving examples of Toastmasters activities, and an overview of the Toastmasters clubs. As part of the pure Toastmasters example, there was a prepared speech with its evaluation and a session of table topics. For the prepared speech, the TM group decided to use one of my previous Competent Communication speech, which had an advantage of being a little bit business oriented. Indeed, I used this speech as a teaser and brief overview of IBIS, one of my current favorite topic. I can’t say I am yet fluent in IBIS, but, the advantage of knowing a topic that only few used to practice is that there is a lot to say. On the other side, this advantage can be a problem too.
The main two objectives of all the prepared speeches at Toastmasters (that should apply to all speeches generally too) are that they should be well structured, with an introduction, a body, and a conclusion; to stay within the 5 to 7 minutes length (at least for the CC speeches). I am not going to dive into the details of how the objectives were achieved (or not), but I will rather go in the way I prepared the speech.
I like to put everything on paper, being virtual or real. So, the first step was to split my preparation sheet in three parts : Introduction, Body, and Conclusion. Then, I wrote what I wanted to explain in the introduction, such as why the people in front of be should be interested in the topic and what they would takeaway, introducing the topic. Then, going into the details of the topic, in this case, the IBIS notation and how to use it. Finally, concluding with the main benefits of the notation and an example of how to use it.
At the end of this first step, I ended up with about 5 pages of text, which is obviously too long. As an average, a good speech rate is around 120 words per minute. Having this number in mind, I started removing content that was not absolutely necessary to the understanding of the topic. That is a difficult part, as you have to put yourself in the shoes of the audience and ask yourself whether a piece is required or not. The advantage with this technique is that it helps going “straight to the point”. At the end, I had only 2 A4 pages remaining, which was reasonable.
Obviously, one does not speak like we write, thus, it is not important at all to spend lot of time on the correctness of the text, nevertheless, what I usually do anyway is to read out the text several times. It has the advantage to give a first timing. And, the more I read it, the more I adjust the wording and expressions, whichI write in the text. Though, the speech is not natural. It is still a text read out, not a speech.
After that step, I am able to extract, and write separately from the text, one and only one word, for each smaller part of the speech, such as the intro, the examples, some part of the body, and the conclusion. From that point on, I only use this list of words for my rehearsals. These words are enough for me to remember the background text, which allows me to get free from the original text. Repetition after repetition, the speech becomes natural, some words only used for texts get replaced by oral words. Of course, the speech will not be exactly the same each time. Some variations will occur, but, the essential of the message is kept.
Another advantage of giving speeches of 5 to 7 minutes is that you can repeat a lot of times. I don’t remember how many times I repeated the speech, but, when you start having the speech completely in the head, it means that you are getting fluent with it and you can consider you are ready to give the speech in front of the audience. Additionally, you realized that you no longer need the notes, and that not focusing on the phrases you have to say will help you using other techniques in your speech, such as vocal variety to emphasize some words or phrases, or the eye contact. Lastly, and especially if you are not used to speak in front of people, repetition will help having self-confidence, decreasing drastically the stress on stage. Arriving at this point, I was ready to deliver my speech.
To summarize, here are some key points of what I do to prepare a speech :
- After documenting the speech’s topic, write down the text of the speech
- Repeat the text by reading it loud
- Summarize the speech using one word per part. I would say maximum 10-12 words for a 6 minutes speech
- Repeat, repeat and….repeat, staying away from the notes more and more.
- Once comfortable, deliver the speech.
- A last trick : I usually spend (and I also read something similar) an average of 1 hour per minute of speech. For the speech mentioned in this post, 6 minutes of speech means around 6 hours of preparation.
How many meetings do you attend ?
From that number of attended meeting, do you read the minutes you receive ?
Then, imagine that the meeting minutes are sent several days after the meeting took place, as a e-mail’s attachment. Chances are that you will not even open that attachment, and process this e-mail, by either deleting it, or moving it in one archive or another.
The goal of a meeting minute, except keeping what was discussed during the meeting on virtual paper, it is also to follow-up on action items, or as a reminder for the participant that something has to be done. Rarely it is used to come back to see who decided of what, and when. For that purpose, I often see a manager maintaining a register of all the decisions and actions in a big Excel file, growing to hundreds or thousands of lines, that even the manager can’t find anything in it.
But, the real problem is that the reasons why actions or decisions were taken are completely missing from the minutes. And, when time comes to blame someone because a previous decision appears to be a wrong one, that same manager will search in the thousands lines to find the guilty guy, but he will never find the reasons and the context of that decisions. They are lost forever. So, we end up with the kind of following statement in a meeting minute :
|Subject || |
Type (Information, Action, Decision)
|To implement a SharePoint Content Type Hub for the intranet in order to categorize the content. || |
The problem with such statement in meeting minutes and the contained decisions is that once written, they are completely separated from their context. As we know, contexts change, making the original decisions obsolete and wrong. Unfortunately, unless the meeting minute is written with a lot of details, the importance of the decision’s context will be forgotten. Additionally, in order to ease the reading of the minutes, items in meeting minutes tend to be short and rather dry, omitting many elements, regardless of their importance, and therefore opening the door to interpretation. Text is just too linear to describe correctly a reasoning or the different explored paths to the decision. Finally, the interpretation will occur at least twice, the first one at writing time, and then when reading the meeting minutes.
In that previous example, unfortunately, the implementation of the SharePoint Content Type Hub didn’t deliver its promises and several month later, looking at the meeting minutes, one discovered that the decision to use a Content Type Hub was taken by the poor Charlie Crews who is now in trouble to justify this decision. Obviously, nobody remember why this decision was taken and the discussions that took place before stating this in the minutes.
So, the question here is, can we avoid this kind of situation, and how this can be achieved ?
Since a year, I started working on ways to capture the reasoning behind decisions made by a group of people or, just to write down all the elements before taking myself a decisions. I am a big fan of the pen and paper way of writing the notes, but, when it is is time to share it with others, the only way to get the same understanding from all the people is to share the same notation. For that purpose, I discovered a little more than a year ago the IBIS notation, that I used for my notes. The good point of this notation is its ease to model the decision making process because of its simple notation, and also the fact that you absolutely don’t need any software to use it. Indeed, a pen and paper do the job well. Also, because its simplicity, there is no need to learn during several days how to understand the different element and icons of the notation, they are pretty straightforward.
I don’t want to enter into the description of the IBIS notation element, but, rather, demonstrate how the example above could be addressed using such technique. Also, I would like to emphasize that it is only an example which does not, even if taken from a real project example, describe the real element or argumentation of any decision of that project. In other words, the goal is not to discuss whether the pros and cons of using a SharePoint Content Type Hub are correct or not in the example. And, to end the “disclaimer”, I am still improving my usage of the notation, so, what is shown below may not be exactly in line with IBIS and dialog mapping (which is another further step in practicing IBIS).
Back to the meeting minutes problem, here is an example of how the decision could have been modeled :
Again, this model may not be complete, but, it gives an idea of how a decision could come up. First, on the left-most end, what is called the “root question”, or, in other words, the question or the problem that needs to be answered. In our example, it is “What is the best way to apply metadata to documents ?”. When debating of that question during the meeting, several answers will be given by the different participants. Each of these answers have benefits, and, on the opposite, drawbacks. All of these elements are also gathered and linked to their related answers. As an example, “not using Content Type at all” also means that “no standardization of metadata” or template is possible.
Are all of these arguments valid ? Well, if there is discussion about an argument, it also has to be present in the diagram, as, again, one of the goals of the diagram is to be transparent and to show when there is disagreement. Another positive point is the neutrality of the diagram, as there is no name associated to an idea, argument or question. Which means that it puts all the participants at the same level.
Then, for one question or problem, several ideas or answers are provided. And, for each of the ideas, pros and cons are also captured on the diagram, but, yet, the question is : how does it help in taking the right decision ?
As mentioned earlier in this post, it is important to keep track of the context and reasons for a decision. That is why, at the bottom of the diagram, there is a question about the solution selection criteria, with answers, that I have put in descendant order of importance : “Centralized Control”, “Search Improvement”, and “Minimal training”. What this describes is that, at the time of the meeting was held, the most important criterion was to have a central place for the management of the Content Types.
Then, instead of sending a word document containing the meeting minutes with context-less decisions, sending the map of the meeting will have the following advantages :
- Even people that are not familiar with IBIS can understand the simple icons and notation
- People can also easily understand why such or such decisions was taken
- Meeting’s participant should not worry about their association with arguments
- Afterwards, if the decision appears to be the wrong one, a good part of the analysis has been done and don’t need to be done from scratch to find a good alternative. Only a review of the existing analysis can be done in order to update the selection criteria, pros, and cons and potentially new ideas.
6 months later, when everything went bad, coming back to this kind of meeting minute will show and demonstrate that the context or the rationale of that decision. From that, either it will be discovered that the decision was not the worst one, or, that environment changed as well as the requirements, leading to another decision to be taken. Another benefit is that Charlie Crews does not appear in the diagram, which means that decision was taken (normally) collegially.
The 9th of december last year (announcements on Scott Hanselman blog and the .NET foundation), Windows Live Writer became Open Live Writer and at the same time became open source through the .NET Foundation. For different licensing and complexity hurdles, several features were removed, and, for the time being, the team is focusing on Windows 10, but it works under Windows 8 for me without problems.
What a good job they did !
That said, as I am working on several computers and actively using OneDrive, I was used to have the local draft folder on OneDrive too. Thus, I was wondering if it was possible to use the same trick as WLW 2012, and adding the PostsDirectory key in the registry in order to set the draft folder.
For Windows Live Writer 2012, the PostsDirectory registry key was under HKCU\Software\Microsoft\Windows Live\Writer .
But, first because it is no longer Microsoft providing this very useful tool and also because it is no longer par of the Live Essentials suite, the registry key had to be added in another location.
Simple things are always the most efficient, there is no need to try long time or search for hours where to add the key, it is simply under HKCU\Software\OpenLiveWriter . Then, restart OLW, if needed, in order to reload the new value of the parameter, and the drafts will be saved in that new location.
Oh, maybe one important thing : if you get your OneDrive files from another computer through OD synchronization, make your drafts “Available Offline”. Indeed, for me, OLW was not able to see the drafts until I made them available locally.
After the install of the Zachman Framework MDG Add-in for Sparx Enterprise Architect, there is an issue when we want to open the provided sample, ZF Example.
EA complains that it "couldn't lock file".
One way to successfully open this file is to run EA as an administrator, but, it is not the most convenient way. It is better to open windows explorer, select the "Program Files (x86)\Sparx Systems\MDG Technology\Zachman" folder, and change the permissions on it. Give "Full Control" to all the users, and it will enable the opening of the example.