Why’s my GIS so slow?
This was the overriding question presented to geoworx in 2019. Virtually every engagement undertaken came back to this fundamental problem.
We all know it, Esri can be slow. People seem to love to blame the software. Whether it’s the NZ GIS Geeks Facebook page or a Regional User Conference, there are usually people having that quiet (sometimes more vocal) discussion about why Esri is slow. I guess if you pay for software, then you have the right to expect that it should perform better.
I have a passion for proving that it’s not the software that’s at fault, well most of the time. Yes, sometimes, I get stumped and have to say – maybe the next version will fix it. But most of the time I can significantly improve the performance for the organisations that choose to use geoworx.
Slow GIS implementations are, let’s face it, frustrating. But does it cause you deeper concerns? In all honesty I’ve had conversations with people who have lost sleep over this issue, worried that they need to find budget for more equipment or more staff. These are the intangible fears that people face and often do not discuss.
It is in this situation where geoworx can bring value. Usually, the problem is more to do with either the IT infrastructure (hardware/network) that underpins the GIS, or the corporate policies that define the corporate usage of a system within an IT context. Sometimes the latter drives the former.
Often there is a gap, in either communication and knowledge, between the GIS team and the IT team. Each will describe their requirements, and my experience shows that they are often mutually exclusive of each other. The immediate thought is that the problem is going to be complex and expensive in terms of time, money or effort. Is it just better to live with a system that is slow but generally works? No!
Faster, more efficient and effective GIS systems:
build trust and acceptance with your users,
reduce frustration for the whole GIS team and downstream users
allow the business as usual and mundane tasks to be optimised
in turn, the team can start to deliver more of the exciting aspects of GIS that we aspire to...
If only we didn’t struggle with the mundane? Where are you on this journey? Are you stuck in the “it’s too slow” camp or are you approaching the “we’re doing more with less” situation? If the former is a one and the latter is a ten, then what is your score? Be honest with yourself. Start 2020 by reflecting on it. If you’re less than an eight, then geoworx can help your journey.
The situation may not be as bad as you may think, some consultancy, coaching and experience may be all that is needed. geoworx has a vision of educating and inspiring, using experience gained so that you and your team can solve these problems for yourselves. I'm happy to get my hands dirty and do the work for you, but the value of having a coach or mentor that is accessible to you means that you can do more for less. geoworx is not here to sell software or sell licenses, although that may be an outcome sometimes. geoworx gains no advantage from you buying more, and we’re not afraid to tell you that you have too much or too little! geoworx exists to promote and coach the use of the best practices. GIS training in NZ is generally excellent, and geoworx will always recommend appropriate formal courses. Our interest is in taking that generic training, or university course, and making it useful where the rubber hits the road.
Generally, teaching of the wider IT considerations is missing from many courses. It is this content that makes the GIS successful. I have been GIS user, but I also have wide-ranging experience of Enterprise IT and making GIS work in that space. I’ve seen the good, the bad and the ugly – it’s that experience that can help you grow in your practice, and help you to improve communication with your supporting IT providers. geoworx is the bridge that crosses the gap.
Let me paint a picture. A few years ago, I had the pleasure of mentoring a GIS newbie in a Central Government GIS team. It was a small team, and they were struggling with producing paper maps from desktop software. This was at a time when server technologies and ArcGIS Online were mainstream. The team was overwhelmed and couldn’t look beyond producing PDFs.
A discussion with the team highlighted some internal IT policies:
All data had to maintained on a file server.
Local disks on all machines were locked down, and there was no way to store data there
Everyone in the organisation had the same machine, which was below the specification required by ArcMap (this was before Pro)
Anti-virus checked every file read from remote servers.
Given the nature of data security in that organisation, the policies are more than valid. They need to be respected. The impact of these policies though was a very slow ArcGIS environment, where there was no hope of meeting the tight timescales that internal customer required, and the team, therefore, appeared to be failing.
My client had, therefore, built a business case for recruiting an additional team member. The fear was presenting this to management.
During a follow up mentoring session, I asked for their assigned IT support person to attend as well. We sat and talked about the situation. I mentioned that the team were ‘developing data products’ and had inadequate resources to be able to do so. The keyword here was ‘developing’. The face of the IT support guy lit up, and he asked: “well why didn’t you ask for a development machine?”. In many customers, I have seen beefy development servers provisioned. In this organisation, they made a workstation available. The workstation was significantly above the recommended ArcMap specification.
Further, we were able to review the policies described above:
We agreed that data products must always reside on the file server, but working datasets were acceptable locally.
The workstation had several Solid State Disks which were available for local storage (as it was a development environment). Local data always out-performs remote data.
The organisation recognised that some use-cases required specialist hardware
Anti-virus scanned inbound data from the file server. There was no scanning of data that was being ‘developed’. The processing was, therefore, less impacted by anti-virus.
Generally, the increase in memory and better CPU make a significant difference. The use of Solid State Disk is a game-changer and its use is critical to highly optimised operations.
In this situation, the new machines and the compromises relating to policies led to a better GIS workflow, and the team were able to deliver more than they needed to. They could respond in double quick time, this allowed the team (without the extra team member) to move towards trialling ArcGIS Online, the results of which offered additional benefits. They were then able to implement ArcGIS Enterprise as well.
The overall cost was for a new development machine, in the order of a few thousand dollars, and not the high tens of thousands of dollars (per year) for an extra team member. It’s these opportunities that geoworx can help you explore.
Let me paint another picture. I had another customer that used a Land Use dataset. When they clicked on a polygon, in a web map, that represented a vast area of the Canterbury Plains. All they wanted was a popup to show the attribution, but that popup action took 16 seconds. That was justifiably unacceptable. To illustrate what was going on, I used Google Chromes developer tools and grabbed the JSON (text) response from the identify operation. I pasted that into a word document in Calibri size 8. The list of all the vertices, for the outer edge, and all the inner (doughnut) rings, ran into many thousands of pages of A4.
I asked if it was acceptable to send this document by email and wait more than 16 seconds, and of course, the answer was yes. However, the non-functional requirements for the GIS had stated that all responses must be sub-second, and so, from the GIS, this outcome was ‘unacceptable’.
It’s easy to blame the software, but the data queried was at fault. There is a tendency in New Zealand to take a dataset captured at, say, 1:10,000 scale and use that at all scales. There’s no kind way to say this, but that is just bad practice. The following summarises a third-party discussion on this topic:
“Sometimes, GIS data contains an excess of detail or spatial information than what is needed for the scale of the map being prepared. Generalisation is the method used in GIS to reduce detail in data. For example, a small scale map of the United States does not need detailed coastlines or a map of California does not need to show every road in the state.
Generalisation can be achieved by removing detail, such as only showing major roads, showing only the boundary of a state instead of all the counties. In GIS generalisation is also used to smooth out lines, removing small detail such as the nooks and crannies of a coastline or the meanderings of a stream.
Since detail about a geographic feature is simplified during generalisation, generalised data is less spatially accurate. Those using generalized data to calculate length, perimeter, or area will incur errors in the calculations.”
Data needs to be maintained and manicured to fit the various requirements of the system. Esri provides the tools for this. After some careful consideration, the customer was encouraged to use a 5km by 5km grid to ‘tile’ the data, rather than generalising it. This process reduced the size of each polygon while still providing quality. Identify operations were configured to use this dataset while draw operation used the original dataset. The identify operation dropped from 16+ seconds to a fraction of a second.
In this situation, there was no requirement to change any infrastructure. Only an understanding of the data and the impact of that data on the IT infrastructure was needed. Minor changes made a significant difference to the user experience of the system.
These are two of the more simple situations that I have tackled over the years. There was no point in losing any sleep over either of them. Just some time with someone with the experience and knowledge of both GIS and IT was required and the result was a low-cost solution.
Throwing more money, CPU or more people at a problem is not always the solution. Indeed, that can make the problem worse. geoworx seeks to coach and mentor in all its engagements. I will identify the problem, discuss it with you, give you options on how to resolve it, and then offer services to remedy it, or support your team to remedy it themselves.
During 2020 geoworx is looking to add more capacity to the team. If this blog has resonated with you, then feel free to contact us. Scott is available on firstname.lastname@example.org or 027 523 8060.
If you want help, then I would be happy to discuss the geoworx approach, provide indicative costs, and potential timescales.