33

In the context of a different question, we encountered a deployment performance degradation on EU5 of between 6000% to 10000% using Metadata-API. This means that work (saving a single APEX file to the server) done usually in 14 seconds took between 950 and 1500 seconds. This situation persisted for some days during the last week and as a result our development work came to an halt for a couple of days.

I guess the entire EU5-Pod was affected for this period, since I was able to perfectly reproduce it on two production Orgs and a couple of developer Orgs. Also other users reported it later, too.

My understanding is, that issues will be reflected transparently at trust.salesforce.com - only that seems not to happen always:

enter image description here

When I can't see an issue there, next thing I do is to feel responsible myself checking for possible reasons on my end (like bad test-classes, bad setup, etc). After investing some hours, it became clear the server caused the situation since others reported the same problem (here and on twitter).

I would be interested what kind of issues or which severity are we supposed to see at trust.salesforce.com and in contrast which issues we should expect not to see there? I found no documentation about it.

As something similar happened in March 2015 for CS14 I assume that everything development-related is unlikely to be shown. This is sad twice, since we have no chance to interact with the Salesforce support nor even to report those kind of issues.

To fire always the known support-communication substitutes like twitter to check if it's already on the radar at Salesforce is time consuming and not very productive. Also it's not very nice and I feel unhappy about my own complaints - even if they might be justified in the end.

So I would like to learn how you deal with such situations and how I might handle it better next time.

Community
  • 1
Uwe Heim
  • 28,350
  • 18
  • 115
  • 283
  • 3
    I have also had the experience of contacting Salesforce Support to be told that there is a known ongoing issue with our pod, and never seeing anything on trust confirming that. – Jeremy Nottingham Jun 22 '15 at 13:17
  • 2
    Hi Uwe. Just so you know - you are not alone. My colleagues and I have been experiencing exact same issues with saving files via Metadata API ever since sandboxes got migrated to Summer'15. trust.salesforce.com does report performance degradation from time to time (usually after the fact), but after it got marked as resolved the issue soon comes back again. The only working workaround we found is to use Tooling API (instead of Metadata API) whenever possible. Removing all managed packages from Dev Sandbox also helps, but nowhere near as effective as saving stuff via Tooling API. – gaiser Jun 22 '15 at 13:24
  • 1
    I don't know the answer, but I see the same things. Last week on EU5, I found deployments taking up to an hour in a "pending" stage that I had never seen before. Got the brush-off from support and nothing but green on trust.salesforce.com.

    A while back communities were down for days, and still all green. Also no useful response from support until over a week later.

    I'd love to know what SF have to say about it... trust.salesforce.com inspires anything but trust, IME. Doing consultant development work, it puts you in a tough position with clients.

     – Aidan Jun 22 '15 at 14:54
  • Thanks @gaiser, Tooling API is no option for our use-case. We are working on clients production Orgs and are in the urgent need to quickly deploy updated files. Multiple times per day. So there is just the good old Metadata-API, which can do this in about 14 seconds for one file (assuming healthy Salesforce-Pod, light tests, fast IDE). There is no way to use Tooling-API to deploy APEX on ProductionOrgs at least as to my knowledge. – Uwe Heim Jun 22 '15 at 15:01
  • 1
    @Aidan that's it: we want to deliver excellent consulting and development work to make sure that clients love Salesforce. In case of such issues we cover for all the loss of working-time and extra effort. In the end, it's not the fact that such issues happen. They may happen. But the fact that there is neither a communication channel nor something on trust.salesforce.com gives us a very hard time... – Uwe Heim Jun 22 '15 at 15:05
  • 1
    @UweHeim Have had similar experiences myself while working in orgs for clients. It's very difficult to justify charging them for extra time while I sit waiting for a test to finish running or for something to deploy. I try to make the time productive doing other things, but as you say, when it persists for a couple of days, it can become very frustrating when others are anxiously waiting on you to finish your work. – crmprogdev Jun 22 '15 at 15:39
  • 1
    This is thrilling us in different pods (eu5, cs18, cs17) and this forced us to use tooling API. – Enrico Murru Jul 16 '15 at 14:01

2 Answers2

3

We've had multiple instances where Salesforce was either extremely slow or unavailable to users - i.e. dropping 3 out of every 4 packets - while trust was indicating that everything was fine. After talking with support, I was told that trust only monitors database performance and other issues such as networking would not be included unless they were extreme.

It was recommended that we use a service like thousand-eyes to monitor networking and if we encountered issues to log the traceroute (instructions here: https://help.salesforce.com/apex/HTViewSolution?id=000025283&language=en_US) and open a ticket and add those logs as an attachment.

brezotom
  • 1,077
  • 17
  • 43
  • It is hard to see what "trust" really means. I'm looking for a statement from Salesforce where they say: this kind of issue is reporter there and that is not. Right now (but this is based only on hearsay) it feels a little bit like they just report what they want - or maybe use trust as a kind of proof, that the service runs perfectly most of the time. This seems to be true from a business users perspective - but not from a developers perspective. Timecritical huge projects needs to take possible platform lags into account. APIs on sandboxes may perform very differently every day. – Uwe Heim Feb 02 '16 at 21:05
  • Here's an article that explains it a little more - http://www.appneta.com/blog/top-5-salesforce-performance-problems/ . I doubt that they would give any statement saying such, but it makes sense that they would only monitor their internal systems, as it would be difficult to monitor the actual user experience – brezotom Feb 02 '16 at 21:21
  • Yes, I perfectly understand that it can't be expected to see an end-to-end performance anywhere. However if 99.9% of the Salesforce-UI (even data-intensive stuff) is very very fast AND at the same 99.9% of Metadata-API plus Tooling-API is reproducible very slow (and I mean 1000% to 10000% slower than usual) it speaks for itself whether to blame the connection or the server, I guess... – Uwe Heim Feb 02 '16 at 21:42
1

I have faced similar issue and here is was support shared with me:

They told me that it was due to momentary perforce degradation. When i asked about why its not captured on trust.salsforce.com they told me that the duration was very small and so was not captured on trust.salsforce.com.

On trust.salsforce.com they have mentioned:

Performance Issues

The indicator means that the average transaction speed for an instance has exceeded 0.5 seconds for 2 minutes or longer.

The performance monitor reports at one-minute intervals the average time it takes to respond to user requests for each instance. If it reports an average time that is greater than 0.5 seconds, the Site Reliability (SR) team investigates the issue to verify that multiple customers are affected. If the issue affects only a single customer, then SR will ensure that salesforce.com Customer Support has the information required to address it for that customer. If a multi-customer degradation lasts for more than 2 minutes, the indicator is posted with a status message detailing the starting time of the degradation. Our target is to post the status message to the Current Status report within 5 minutes of validating that multiple customers are affected, and to update the message every 30 minutes until the issue is resolved. SR will report that the issue is resolved once the performance monitor reports an average degradation time of less than 0.5 seconds for a one-minute interval. Once this occurs, SR posts the incident report to the Service Performance History matrix and replaces the indicator with the indicator.

So, is possible if there are multiple spikes of 1.5 minutes (aprox) then we may face this issue but its not posted on trust.salsforce.com.

How to deal with such situation:

In my case I was calling SFDC api and getting response that server is not available. So I just use the re-try mechanism. after 3 tried I stooped my processing and retry after 12 hours.

AtulRajguru9
  • 9,110
  • 3
  • 28
  • 66
  • My observations exceed the threshold you mentioned above by FAR and over multiple days without being reported. My guess is, that "The indicator means that the average transaction speed for an instance has exceeded 0.5 seconds for 2 minutes or longer." is NOT used for MetadataAPI and ToolingAPI. My gut feeling is, that such developer API issues are reported either just rarely or not at all. I'm not even sure if they are considering such an request as "Transaction"... what I'm still missing is a feedback from one of the Salesforce employees around here to clarify this uncertainty. – Uwe Heim Jan 12 '16 at 11:52
  • 1
    Basically you need some one from SFDC Team and tell you how exactly the metadata API are handled and in the context of performance degradation. If that is the case then i will delete my answer so that this question will appear is unanswered questions list and may be it would be seen by some one from SFDC team? – AtulRajguru9 Jan 13 '16 at 04:01
  • can't get that answer via support. without Premier Plus Support, they wont talk to you about that (because developer support is out of scope). – Uwe Heim Jan 14 '16 at 15:58