26 Replies Latest reply: Jul 12, 2013 10:26 AM by Jeffrey Vermeire RSS

    SR1 - SVN

    James Gilpin

      Has anyone downloaded QV11 SR1 and played with the SVN interface?

       

      James

        • SR1 - SVN
          Edgar Kech

          As I want to integrate QV with SVN, too, I will test it next week. Qliktech said, they did some changes in the XML-structures (we had some problems with IR and lost settings when using XML-Files / -prj-Folder). Hope, they fixed this kind of problem and we get a real seemless integration...

          regards,

          Edgar

            • SR1 - SVN
              James Gilpin

              Here is what I found (so far)

               

              Pros:

               

              1)  management of the -prj (including creation) is handled by the desktop client

              2)  .qvw is NOT stored in SVN; when you checkout the working copy the .qvw is created from the -prj files

              3)  add project to SVN and get project from SVN work well.  however, you will need to setup the project folder structure (branches, tag, trunk) on your own.

               

              Cons:

               

              1) Requires an external merge tool.  There are some settings to point to the tool of your choice.  I am still trying to get kdiff3 and tortoise to accept the incoming file names.  If I drop out to windows explorer, the standard TortoiseSVN plugin will start up the merge tool correctly.  I prefer kdiff3 and it handles the required changes.  Sure would be nice to have visual tool that showed the difference instead of inferring the placement of objects.

              2) Documentation.  Or the lack thereof...

               

              Some recommended steps to use the interface:

               

              1) Always 'Get Latest Version' to avoid conflicts, prior to making changes.

              2) Be prepared to resolve conflicts externally.

               

              More to come, I suspect...

               

              James

                • Re: SR1 - SVN
                  James Gilpin

                  Update

                   

                  After extensive testing over the past two days, I am even more confused regarding the ability to use a version control system to track changes to QlikView documents.

                   

                  Base Assumption:

                   

                  Never store the .qvw in the repository.  You cannot view the changes to the binary and the resulting repository growth can be overwhelming.  Which means the QlikView desktop is require to get the latest version of the .qvw -prj files and rebuild the .qvw document.

                   

                  Findings:

                   

                  The QlikView desktop will successfully rebuild the .qvw based on the objects stored in the -prj folder and placed in the repository.  What is upsetting is that when the document is rebuilt, certain values stored in the .xml objects are CHANGED by simply rebuilding the document.  If you reload the data, other values are also changed.  The result:  the -prj folder now contains .xml files that differ from the repository.

                   

                  So what, you ask?  Is that not the point of the repository?

                   

                  Take the typical use of a source code repository...

                   

                  Dev creates the .qvw document and uses the QlikView desktop to 'Add Project to Source Control', which adds the -prj folder into the repository

                  User 'Get[s] Project from Source Control' which creates a working copy of the -prj folder and rebuilds the .qvw document.

                  Test Server 'Get[s] Project from Source Control' which creates a working copy of the -prj folder and rebuilds the .qvw document and the reloads the data.  (QVS, Access Point, auto build tool)

                  User reports an issue and requests a modification

                  Dev uses the QlikView desktop option to 'Get Latest Version' of the -prj folder (best practice to stay up to date, before starting new changes.)

                  Dev makes changes and uses the 'Check in Pending Changes' into the repository

                  User gets the latest changes, and finds that conflicts have occured (remember, the QlikView desktop has rebuilt the document and .xml files in the -prj folder used by User are now different than the repository)

                  Test Server gets the latest working copy, finds that conflicts have occured....

                   

                  Question:

                  Does this happen using the TFS interface?

                   

                  James

                    • SR1 - SVN
                      John Trigg

                      Why are you placing the QVW in the repository?  The whole purpoe of the integration is to move the component files (found int he PRJ folder) into the repository and manage from this definition.  The goal has never been to version control the QVW given its potential size (with the data).

                        • Re: SR1 - SVN
                          James Gilpin

                          Perhaps I was not clear, the base assumption I made is in total agreement with your statement.  I am NOT storing the .qvw document.  Only the files that are stored automatically by the QlikView desktop.

                           

                          It is these files that are in conflict between the different working copies.

                          • SR1 - SVN
                            Brian Garland

                            I had made the assumption that the .qvw would be stored in the repository but with the data stripped out, sort of how Publisher opens the .qvw without data when it performs a reload. We're going to go ahead and test it anyway to see if it will work for us.

                            • Re: SR1 - SVN
                              James Gilpin

                              John,

                               

                              I reread the above post and realized the wording did not reflect actual process.  It has been edited to more accurately reflect the actions taken by each individual in the process.

                               

                              thanks,

                               

                              James

                              • SR1 - SVN

                                Hi John,

                                Echoing Brian Garland's comments - I would definitely like the ability to save the QVW binary (deflated) in the source code repository. I understand that theoretically that isn't really necessary. But I've found plenty of times where it's been a "last resort" to have a version-controlled binary tucked-away in a safe spot.

                                 

                                A feature request I would look for is a toggle switch that allows "Include Binary In Repository." And naturally I would look for the feature to automatically reduce all data from the QVW.

                                 

                                Tks,

                                Bill

                              • Re: SR1 - SVN
                                John Trigg

                                James

                                 

                                I want to ask some questions of your steps (which I've nunbered for ease of reference). Can understand #2 & #3, but is the user going to test the document on the test server or locally? Lets say its tested locally by the user and posted on the test server.  So developer does some work and checks in changes.  User gets the latest and finds these changes; doesnt seem unreasonable that their version is going to be in conflict. Now one best practice maybe for the user to clear out their PRJ folder before doing get latest, if they are not doing development.  Same maybe true for the test server.

                                 

                                I guess what I am looking for guidance on is what do you expect the user and test server to be faced with in steps 7 & 8?

                                jmgilpin wrote:

                                 

                                Take the typical use of a source code repository...

                                 

                                1. Dev creates the .qvw document and uses the QlikView desktop to 'Add Project to Source Control', which adds the -prj folder into the repository
                                2. User 'Get[s] Project from Source Control' which creates a working copy of the -prj folder and rebuilds the .qvw document.
                                3. Test Server 'Get[s] Project from Source Control' which creates a working copy of the -prj folder and rebuilds the .qvw document and the reloads the data.  (QVS, Access Point, auto build tool)
                                4. User reports an issue and requests a modification
                                5. Dev uses the QlikView desktop option to 'Get Latest Version' of the -prj folder (best practice to stay up to date, before starting new changes.)
                                6. Dev makes changes and uses the 'Check in Pending Changes' into the repository
                                7. User gets the latest changes, and finds that conflicts have occured (remember, the QlikView desktop has rebuilt the document and .xml files in the -prj folder used by User are now different than the repository)
                                8. Test Server gets the latest working copy, finds that conflicts have occured....

                                 

                                Question:

                                Does this happen using the TFS interface?

                                 

                                James

                                  • Re: SR1 - SVN
                                    James Gilpin

                                    John,

                                     

                                    If neither the user nor the test server has made changes, no conflicts should occur.

                                     

                                    But it appears that the desktop is making changes to the .xml files based on the presence of data.  Sort criteria, present/not present, etc.

                                     

                                    James

                                    • Re: SR1 - SVN
                                      James Gilpin

                                      John,

                                       

                                      In step 2, the user gets a copy of the -prj as it sits in the repository.  As part of this process the desktop application recreates the .qvw document (as expected).  The process has the desktop application store a blank .qvw document and then applies the -prj xml files.  Once the -prj xml files have applied, then user is presented with the layout, script, etc. for review.  This restored copy of the .qvw is NOT saved until the User tells QlikView to save the document.

                                       

                                      IF the user SAVES the document at this point, the resulting -prj xml files are overwritten.  Even though the user has not made any changes some of the -prj xml files are different.  In my case, 9 out of 50 are different.  The next time the user issues the 'Get Latest' command a conflict is detected.

                                       

                                      I thought maybe it was due to platform, 32bit/64bit, or possibly different user preferences.  But this occurs between two different working copies of the -prj folder on the same machine.

                                       

                                      James

                                        • Re: SR1 - SVN
                                          John Trigg

                                          So have spent some time playing around with this today and cant quite replicate your conflict scenario, but put that down to user setup error on my part.  I suspect a date/time stamp is updating and forcing the conflict though I cannot yet confirm that.  Will do some more poking around and talk to the appropriate developers to get more insight.

                                           

                                          Cheers

                                          John

                                            • Re: SR1 - SVN
                                              James Gilpin

                                              John,

                                               

                                              Thanks

                                               

                                              Should you need an example document, give a shout and I will redact this one I have been playing with and post it....

                                               

                                              James

                                                • SR1 - SVN
                                                  Rob Wunderlich

                                                  James,

                                                   

                                                  I want to confirm your scenario. The user only does:

                                                   

                                                  1.Open

                                                  2. Reload

                                                  3. Save

                                                   

                                                  And that creates changes in some of the object.xml files? I would expect a conflict in the AllProperties file, but no others. Any column sorting, object moving/restoring before save will cause change as well because those are layout changes.

                                                   

                                                  I'd be curious to know what files and properties you are getting conflict on.

                                                   

                                                  -Rob

                                                    • Re: SR1 - SVN
                                                      James Gilpin

                                                      Rob,

                                                       

                                                      It is not quite that simple.  This appears to be an issue with how the .qvw is rebuilt after getting the -prj from the svn repository.  When the desktop application reads the svn repository it must recreate the .qvw based on the files in the -prj folder.

                                                       

                                                      In my test case, three files are different...

                                                       

                                                      AllProperties.xml

                                                      CH01_391551278.xml

                                                      QlikViewProject.xml

                                                       

                                                      I created a 7zip archive of my test files to show these differences.  (yes, i included the .svn folder... bad developer...)

                                                       

                                                      James

                                      • SR1 - SVN
                                        Torben Seebach

                                        Has anyone managed to move code between servers without committing the .qvw?

                                         

                                        I looking for at method to maintain a central repository, and how this can be moved between sandbox, test and prodution. But I'm puzzled on how to best handle creating .qvw files if a new file is added to the repos.

                                         

                                        I'm disregarding the source control options in qlikview, since are running some very strange commands, and don't allow me to check out dependencies. As far I can tell the buildt in functions are very limited, and don't support a standardlized version control method.

                                         

                                        SVN is super scriptable, so I'm thing of making server hooks that perhaps adds a blank .qvw when a new -prj folder i found.

                                          • SR1 - SVN
                                            James Gilpin

                                            I have Jenkins in place performing builds of the .qvw by use of the 'svn revert', 'svn update', and 'qv.exe /r' commands.  But in my workflow I manually add a blank document for the initial setup.  It would be very interesting to see a script that detected a new -prj folder and copied a blank document into place.

                                             

                                            The next step for our group is to setup a code review process that requires an approved signature for changes to the .qvw document stored in the repository.  Note: this has no bearing on the users, only developers, ie. data source, joins, etc.

                                             

                                            James

                                            • SR1 - SVN
                                              Rob Wunderlich

                                              I build the qvw from -prj using VBScript:

                                               

                                              The CreateQvw() is enough to create the qvw that will then be populated when a user opens it. If you want it to be fully populated call the OpenClose() as well. The qvw name is determined in another process and passed to the routine. Hopefully this will give you a guide as to the steps to take after checkout. Because it's using QlikView, a GUI is required. I haven't figured out how to build the qvw in a headless environment.

                                               

                                              -Rob

                                               

                                              Function CreateQvw(qvwFullPath)

                                                        Set Qv = CreateObject("QlikTech.QlikView")

                                                        Set docObj = Qv.CreateDoc

                                                        docObj.SaveAs qvwFullPath

                                                        docObj.CloseDoc

                                                        Set Qv = Nothing

                                                        CreateQvw = qvwFullPath

                                              End Function

                                               

                                              Function OpenClose(qvwFullPath)

                                                Set Qv = CreateObject("QlikTech.QlikView")

                                                Set docObj = Qv.OpenDocEx (qvwFullPath,0,false)  ' Open the document

                                                docObj.Save

                                                docObj.CloseDoc

                                                Set Qv = Nothing

                                                OpenClose = qvwFullPath

                                              End Function

                                                • SR1 - SVN
                                                  James Gilpin

                                                  Rob,

                                                   

                                                  Do you find that you lose actions/triggers associated with fields by using this method?

                                                   

                                                  For example, when the blank document is created using the xml files in the -prj folder, QlikView is not aware of any fields until the document is reloaded.  The association of triggers/actions to those fields is lost (or appears to be) by creating a blank document.

                                                   

                                                  James

                                              • SR1 - SVN
                                                James Gilpin

                                                Update

                                                 

                                                QV11 SR1 resolved a couple of minor issues, however there are more difficult issues which everyone should know.

                                                 

                                                Issues:

                                                 

                                                1) requires svn 1.6.17 (client)

                                                 

                                                You cannot use version 1.7 of the svn client with the supplied QV SVN provider dll (source control settings).  Version 1.7 svn merge variables (%base, %mine, and %theirs) are handled incorrectly by the QV dll.  Downgrading to svn 1.6 (client) resolves this issue.  However, downgrading and trying to maintain working copies created under 1.7 OR using Tortoise 1.7 and svn client 1.6; NOT recommended.  Best to start over by uninstalling all the svn client software (including Tortoise, VisualSVN, etc.) and reinstalling the versions supported.

                                                 

                                                2) Get Project from Source Control creates working copy, regardless of existing working copy

                                                 

                                                I am being picky, but if the working copy structure already exists...  QV should recognize the working copy and use it instead of creating a new one inside the existing working copy.  (multiple .svn directories cause all kinds of grief).  You can force QV to recognize the structure by copying the SourceControlSettings.ini into the -prj folder within the existing working copy.

                                                 

                                                3) Field Event Triggers can be lost... ARGH! (yes, i opened a support case)

                                                 

                                                QV uses the .xml files in the -prj folder to recreate the .qvw document.  The .xml files represent the pieces of the .qvw.  When you 'get project from source control', QV rebuilds the .qvw document.  Triggers based on fields within load statements are not saved in the .xml files.  So when the .qvw document is recreated there is nothing to use to rebuild these triggers.

                                                 

                                                James

                                                  • Re: SR1 - SVN
                                                    darrinm4

                                                    James, Did you get a response about the Field Event Trigger issue with SVN? I found the same behavior and it's a show stopper for me to use version control.

                                                     

                                                    Is there a work around?

                                                      • Re: SR1 - SVN
                                                        James Gilpin

                                                        No.

                                                         

                                                        We store the empty binary .qvw along with the export xml files

                                                         

                                                        James

                                                          • Re: SR1 - SVN
                                                            darrinm4

                                                            How do you empty the binary .qvw before storing in svn? Is it a manual perging of data in QlikView, save, and syncing file manually to SVN outside QlikView?

                                                              • Re: SR1 - SVN
                                                                James Gilpin

                                                                In the post commit script, the size of the file is checked and the commit is rejected if it is over 1 meg.  This is a basic size - the thought being that most documents would be over 1 meg (with data).

                                                                 

                                                                The developer then has to use the reduce all function, save the empty file, and commit the .qvw document.

                                                                 

                                                                Not ideal, but it works.

                                                                 

                                                                James

                                                          • Re: SR1 - SVN
                                                            Jeffrey Vermeire

                                                            I can confirm this is still an issue in Qv11.2 SR1.  Has anyone found a solution or is there any movement from QvSupport?  This is a pretty decent bug to deal with.  I have a developer ready to pull his hair out since every time he checks-in his changes, POOF!  Triggers are gone.

                                                          • Re: SR1 - SVN
                                                            Brian Vander Zanden

                                                            James-

                                                            I downloaded the latest version of VisualSVN, and TortoiseSVN and am having the merge issue.

                                                             

                                                            Did you have to use release 2.1.11 of VisualSVN Server? 

                                                            I've found athe 1.6.16 version of TortoiseSVN, but not an earlier version of VisualSVN.

                                                             

                                                            Thx.

                                                            Brian VZ