Power tools are fun.
THIS PYTHON SCRIPT HAS CHANGED MY LIFE.
Whipped it out twice this week trying to get code written under tight deadlines.
Having a nearly-instant, low-clicking-around answer to, "Would it probably work in production?" so I can easily watch how every single change to just 1 line of code behaves, before doing a full-on deploy of code that "seems to work" with a proper deployment tool and "Run All Tests," has been AMAZING.
Lucky me, I got to attend Dreamforce again. Here's a summary.
MiscI wanted to share with you some lessons I learned while working on a trigger.
Because a trigger against a "just-saved" record can kick off another "save" operation on that record (typically the point of a trigger), the very same trigger can get re-fired during what's known as the same "execution context."
In Apex trigger programming, it's considered best-practice to make sure that any subsequent re-firings of the same trigger against the same record don't waste valuable CPU cycles when this happens (because Salesforce limits them within an "execution context").
Therefore, when writing a trigger that "does ____ for a just-saved record," it's important to make sure that, at some point, the trigger saves the ID of that record into a "I already did _____ on all of these records" flag that's viewable across all of these recursions (usually a class-level "Static"-flagged "Set<Id>"-typed variable in the trigger handler).
And, of course, you need to program your trigger to pay attention to its own flags and avoid running expensive "consider doing ____" code against records already in that "already-did-____" set of IDs.
The interesting question is: When do you set the "I already did _____" flag?
If one is comfortable authoring / editing the triggers/processes/workflows that are responsible for such impactful mid-execution-context value-changes, sometimes it's possible to refactor them so that they're simply part of the same "trigger handler" code as the one you're in the middle of writing.
You could precisely control the order of code execution "after a record is saved" and author these actions in a way that ensure there will never be any "mid-execution-context surprise value-changes."
That might let you use the more efficient recursion-reduction pattern instead.
Sometimes, though, there's nothing you can do but choose the 2nd option.
Warning: the attached Python script makes you put a password straight into the code and helps you make an end-run around all sorts of change-management best practices.
Basically, if you're not a little horrified to see this script publicly shared, you probably don't understand what it's capable of enough to be using it -- so please dont!
That said, for circumstances in which you were going to ignore a lot of change-management issues anyway, or just want to do a "check only" deploy of code to a Salesforce org, etc., this code basically lets you type a handful of classes / pages / etc. into a Python script, type in your username & password, say which tests you want to run and how you want to deploy it, and see the results really quickly.
The idea is to be almost as handy, to a developer, for small changes, as right-clicking on code files in Eclipse and "deploying" from there -- the problem with Eclipse being that it doesn't have a "Run Specified Tests" option.
At some point I might create something similar to this that lets you log into two Salesforce orgs, download a smattering of code from one to your local hard drive (instead of already having to have downloaded it with Eclipse), and proceed with the deploy from there. That really feels like playing with laziness/sloppiness fire, though (somehow seems to take out the "Would I really have deployed this just with Eclipse, anyway?" factor).
This code is probably best just for "checkOnly=TRUE" deploys. For real deploys, it's probably still best to Run All Tests, and using a "Change Set" is a lot better for anyone else who might have to stumble into your org a few weeks later and see what's been happening as far as code deploys. (And that's just a bare minimum of version control for simply-maintained orgs.)
A few notes on the code:
import os
import shutil
import base64
from xml.etree import ElementTree
import xml.dom.minidom
import requests
import re
def getUniqueElementValueFromXmlString(xmlString, elementName):
#Extracts an element value from an XML string.
#For example, invoking getUniqueElementValueFromXmlString('<?xml version="1.0" encoding="UTF-8"?><foo>bar</foo>', 'foo') should return the value 'bar'.
elementsByName = xml.dom.minidom.parseString(xmlString).getElementsByTagName(elementName)
elementValue = None
if len(elementsByName) > 0: elementValue = elementsByName[0].toxml().replace('<' + elementName + '>', '').replace('</' + elementName + '>', '')
return elementValue
metadataAPIVer = '40.0'
#'''
toOrgLoginEnvelope = """<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><login xmlns="urn:partner.soap.sforce.com"><username>{username}</username><password>{password}</password></login></soapenv:Body></soapenv:Envelope>""".format(
username='username', password='password')
tor = requests.post('https://login.salesforce.com/services/Soap/u/'+metadataAPIVer, toOrgLoginEnvelope, headers={'content-type':'text/xml','charset':'UTF-8','SOAPAction':'login'})
tosessid = getUniqueElementValueFromXmlString(tor.content, 'sessionId')
tohost = getUniqueElementValueFromXmlString(tor.content, 'serverUrl').replace('http://', '').replace('https://', '').split('/')[0].replace('-api', '')
toorgid = re.sub(r'^.*/([a-zA-Z0-9]{15})$', r'\1', getUniqueElementValueFromXmlString(tor.content, 'serverUrl'))
toapiver = re.sub(r'^.*/([0-9.]+)/[a-zA-Z0-9]{15}$', r'\1', getUniqueElementValueFromXmlString(tor.content, 'serverUrl'))
#'''
# EXAMPLE CODE: thingsToAdd = {'classes':[''],'pages':['']}
thingsToAdd = {'classes': {'singCaps':'ApexClass','ext':'cls','toUpl':['OpportunityETLHandler','OpportunityETLTest']}, 'pages': {'singCaps':'ApexPage','ext':'page','toUpl':['OpportunityETLPage']}}
inputstr = 'C:\\EXAMPLEFOLDER\\EclipseWorkspace\\My Sandbox\\src\\'
outpbase = 'C:\\EXAMPLETEMPFOLDER\\temppkgtouploadfromeclipse\\'
outpfiles = outpbase + '\\filesbeforezip\\'
outpzip = outpbase + 'uploadme'
#'''
# BEGIN: Code to create package
if not os.path.exists(outpbase): os.makedirs(outpbase)
if not os.path.exists(outpfiles): os.makedirs(outpfiles)
pkgroot = ElementTree.Element('Package', attrib={'xmlns':'http://soap.sforce.com/2006/04/metadata'})
for folder in thingsToAdd.keys():
innerDict = thingsToAdd[folder]
typesElem = ElementTree.Element('types')
if not os.path.exists(outpfiles+folder+'\\'): os.makedirs(outpfiles+folder+'\\')
for item in innerDict['toUpl']:
membersElem = ElementTree.Element('members')
membersElem.text = item
typesElem.append(membersElem)
shutil.copy(inputstr+folder+'\\'+item+'.'+innerDict['ext'], outpfiles+folder+'\\'+item+'.'+innerDict['ext'])
shutil.copy(inputstr+folder+'\\'+item+'.'+innerDict['ext']+'-meta.xml', outpfiles+folder+'\\'+item+'.'+innerDict['ext']+'-meta.xml')
namesElem = ElementTree.Element('name')
namesElem.text = innerDict['singCaps']
typesElem.append(namesElem)
pkgroot.append(typesElem)
verElem = ElementTree.Element('version')
verElem.text = metadataAPIVer
pkgroot.append(verElem)
with open(outpfiles+'package.xml', 'w', newline='') as fw:
dom_string = xml.dom.minidom.parseString(ElementTree.tostring(pkgroot)).toprettyxml(encoding='UTF-8', indent=' ')
dom_string = '\n'.join([s for s in dom_string.decode('UTF-8').splitlines() if s.strip()]) + '\n'
fw.write(dom_string)
# END: Code to create package
#'''
#'''
# BEGIN: Code to create ZIP
shutil.make_archive(base_name=outpzip, format='zip', root_dir=outpfiles, base_dir='./')
zipString = None
with open(outpzip+'.zip', 'rb') as f: zipString = base64.b64encode(f.read()).decode('UTF-8')
# END: Code to create ZIP
#'''
# TO DO: Figure out how to run several tests. Maybe it's just a comma-separation? I think it it's that in the point-and-click Change Set UI.
#'''
# BEGIN: Code to deploy ZIP
testLevel = 'RunSpecifiedTests' # Most common values will be 'RunSpecifiedTests' or 'RunLocalTests'
deployEnvelope = """<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:met="http://soap.sforce.com/2006/04/metadata">
<soapenv:Header>
<met:SessionHeader>
<met:sessionId>{sessionid}</met:sessionId>
</met:SessionHeader>
</soapenv:Header>
<soapenv:Body>
<met:deploy>
<met:zipFile>{zipfile}</met:zipFile>
<met:deployOptions>
<met:checkOnly>true</met:checkOnly>
<met:rollbackOnError>true</met:rollbackOnError>
{runtests}
<met:singlePackage>true</met:singlePackage>
<met:testLevel>{testlev}</met:testLevel>
</met:deployOptions>
</met:deploy>
</soapenv:Body>
</soapenv:Envelope>""".format(sessionid=tosessid, zipfile=zipString, testlev=testLevel, runtests=''.join(['<met:runTests>'+x+'</met:runTests>' for x in ['OpportunityETLTest1','OpportunityETLTest2']]) if testLevel=='RunSpecifiedTests' else '')
deploytor = requests.post('https://'+tohost+'/services/Soap/m/'+toapiver+'/'+toorgid, deployEnvelope, headers={'content-type': 'text/xml', 'charset': 'UTF-8', 'SOAPAction': 'deploy'})
# END: Code to deploy ZIP
#'''
#'''
# BEGIN: Code to check deploy
checkDeployStatusEnvelope = """<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:met="http://soap.sforce.com/2006/04/metadata">
<soapenv:Header>
<met:SessionHeader>
<met:sessionId>{sessionid}</met:sessionId>
</met:SessionHeader>
</soapenv:Header>
<soapenv:Body>
<met:checkDeployStatus>
<met:asyncProcessId>{process_id}</met:asyncProcessId>
<met:includeDetails>true</met:includeDetails>
</met:checkDeployStatus>
</soapenv:Body>
</soapenv:Envelope>""".format(sessionid=tosessid, process_id=getUniqueElementValueFromXmlString(deploytor.content, 'id'))
checkdeploytor = requests.post('https://'+tohost+'/services/Soap/m/'+toapiver+'/'+toorgid, checkDeployStatusEnvelope, headers={'content-type': 'text/xml', 'charset': 'UTF-8', 'SOAPAction': 'checkDeployStatus'})
#print(checkdeploytor.content)
#print()
print('Id: ', getUniqueElementValueFromXmlString(checkdeploytor.content, 'id'))
print('Done: ', getUniqueElementValueFromXmlString(checkdeploytor.content, 'done'))
print('Success: ', getUniqueElementValueFromXmlString(checkdeploytor.content, 'success'))
print('Status: ', getUniqueElementValueFromXmlString(checkdeploytor.content, 'status'))
print('Problem: ', getUniqueElementValueFromXmlString(checkdeploytor.content, 'problem'))
print('NumberComponentsTotal: ', getUniqueElementValueFromXmlString(checkdeploytor.content, 'numberComponentsTotal'))
print('RunTestResult: ', getUniqueElementValueFromXmlString(checkdeploytor.content, 'runTestResult'))
print('NumberTestsTotal: ', getUniqueElementValueFromXmlString(checkdeploytor.content, 'numberTestsTotal'))
#print(getUniqueElementValueFromXmlString(checkdeploytor.content, 'details'))
# END: Code to check deploy
#'''
The CRMFusion company makes powerful Salesforce-record-editing software called DemandTools.
Below are some screenshots of setting up its "MassImpact" (single-table-editing) module to do a job equivalent to running an "UPDATE...SET...WHERE" DML SQL statement against a traditional database, for data cleansing within a specific single table.
For example, to turn all values of table Contact, field Home_Phone__c that are filled with nothing but 10 digits in a row, no punctuation, into a properly formatted US phone number, you might traditionally use the following Oracle-database-friendly DML SQL:
UPDATE Contact
SET Home_Phone__c = REGEXP_REPLACE(Home_Phone__c,'^(\d{3})(\d{3})(\d{4})$','(\1) \2-\3')
WHERE REGEXP_LIKE(Home_Phone__c,'^(\d{3})(\d{3})(\d{4})$')
In DemandTools, you would set up a MassImpact "scenario" as follows:
P.S. Just for geekiness, and to compare ease of use, here's some "Execute Anonymous" Salesforce Apex code along the same idea.
(Note: not tested at scale. Depending on your trigger/workflow/process builder load, might not actually work since it probably all runs in 1 "execution context" of post-DML CPU usage "governor limits," whereas DemandTools will run in truly separate "execution contexts" per 200 records to UPDATE.)
Map<Integer, List<Contact>> csToUpdate = new Map<Integer, List<Contact>>();
Integer csToUpdateCount = 0;
Integer currentBatch = 0;
List<Contact> cs = [SELECT Phone FROM Contact WHERE Phone <> null];
Pattern p = Pattern.compile('^(\\d{3})(\\d{3})(\\d{4})$');
for (Contact c : cs) {
Matcher m = p.matcher(c.Phone);
if(m.matches() == true) {
csToUpdateCount++;
currentBatch = (csToUpdateCount/200)+1;
if (!csToUpdate.containsKey(currentBatch)) { csToUpdate.put(currentBatch, new List<Contact>()); }
c.Phone = m.replaceFirst('($1) $2-$3');
(csToUpdate.get(currentBatch)).add(c);
}
}
if (csToUpdate.size() > 0) {
for (List<Contact> csToU : csToUpdate.values()) {
UPDATE csToU;
}
}
Quick code dump: if you download all your "report types" with Eclipse, this can help you put together "fresh" copies of them for use with the Workbench Deploy web site (just zip them up). Very handy if you just created a custom field on an object that's involved in dozens of report types.
This is an example of why it's good to be able to work with XML, too, in Python!
Hopefully I can clean up (or delete & replace) this post at a later date.
import os
from xml.etree import ElementTree
import xml.dom.minidom
def stripNSFromET(etRoot):
etRoot.tag = etRoot.tag.split('}', 1)[1] # strip all namespaces level 1
for el in etRoot:
if '}' in el.tag:
el.tag = el.tag.split('}', 1)[1] # strip all namespaces level 2
for el3 in el:
if '}' in el3.tag:
el3.tag = el3.tag.split('}', 1)[1] # strip all namespaces level 3
for el4 in el3:
if '}' in el4.tag:
el4.tag = el4.tag.split('}', 1)[1] # strip all namespaces level 4
for el5 in el4:
if '}' in el5.tag:
el5.tag = el5.tag.split('}', 1)[1] # strip all namespaces level 5
return etRoot
def isTableOfInterest(tableTagText, fieldsDatabaseTableAPIName):
if fieldsDatabaseTableAPIName == 'Contact':
return tableTagText == 'Contact' or tableTagText.endswith('.Contacts') or tableTagText.endswith('OpportunityContactRoles')
else:
return tableTagText == fieldsDatabaseTableAPIName or tableTagText.endswith('.'+fieldsDatabaseTableAPIName)
outpstr = 'C:\\example\\temppackagereporttypes\\'
pstr = 'C:\\examplepath\\EclipseWorkspace\\orgfolder\\src\\reportTypes\\'
fstr = 'C:\\examplepath\\EclipseWorkspace\\orgfolder\\src\\reportTypes\\Contact_With_Campaign_Members.reportType'
metadataAPIVer = '40.0'
fieldsDBTableAPIName = 'Contact'
fieldAPINames = ['New_Field_1__c','New_Field_2__c']
fieldAPINames.sort()
# SOME NOTES OF INTEREST:
# Contact is not only referred to in the normal way, but also often as OpportunityContactRole coming off an "Opportunity." Not sure if ALL such OCRs are actually able to take Contact fields.
# This could make it a bit difficult to decide into what "Section," exactly, to add a new database field when adding it to every ReportType:
# A given "Section" tag within a ReportType's XML file is NOT limited to containing "Columns" elements with the same database "Table" value. For example, Graduate_Admissions.reportType, with a base object of Contact, has a section named "Recruit data" with fields from both the "Contact.Program_Interests__r" and "Contact" tables in it.
# A given database "Table" value can appear in multiple "Section" tags within a ReportType's XML file (e.g. "Contact" fields could be spread across multiple sections).
# CampaignMember often ends up repeated -- once as a top-level, once below itself -- in a ReportType where it is included (with fields duplicated and everything). Good thing I'm not yet adding any CampaignMember fields to reports.
# SOME CODE TO REMEMBER:
# The number of tables mentioned in a ReportType file: len({e.text for e in root.findall('./sections/columns/table')})
# The actual tables mentioned in a ReportType file: {e.text for e in root.findall('./sections/columns/table')}
# The number of "sections" in a ReportType file: len(root.findall('./sections'))
# The labels of the actual "sections" in a ReportType file: {e.find('masterLabel').text for e in root.findall('./sections')}
# Iterating over a dict: for k, v in d.items():
"""
Algorithm per ReportType file, parsed into an XML-ElementTree node called "root":
- Figure out how many 1st-level nodes called "sections" there are.
- For each 1st-level "section" node, if it's "of interest," set it aside as a key to a dict called "d",
and a list of all its relevant child 2nd-level "columns" nodes as the values.
- Loop through "d" and set aside any keys that have the most list-items as a value, of all keys in "d";
set aside a list of any such keys as "topSections"
- If there was just 1 such "top section" key, set that 1st-level "section" node aside as the variable "se"
(This is, visually to the end user, where we'll be adding the new field.)
- If there was more than 1, arbitrarily pick 1 and set that 1st-level "section" node aside as the variable "se"
- Once we've picked a 1st-level "section" node as "se," cache a dict "uniqueTableNames"
of the distinct words appearing among its grandchild (3rd-level) "table" tag values as "key"
and the count-of-table-tag-per-distinct-word as "value."
- Presuming there were any (I guess this serves as a sort of dummy-check for the section),
instantiate a new ElementTree "columns" node called "newColumn,"
append "newColumn" to "se" (make it 2nd-level)
and flesh out "newColumn" with details of the field we want to add.
(When fleshing it out, we arbitrarily pick a value for the 3rd-level "table" tag if "uniqueTableNames" had several keys.)
"""
changedThese = []
pkgroot = None
for i in os.listdir(pstr):
root = None
if not i.startswith('wrt_'):
with open(pstr+i, 'r', encoding='utf-8') as f: # Open a ReportType metadata XML file
root = stripNSFromET(ElementTree.parse(f).getroot()) # Store a reference to the root of the XML file currently open in a variable
d = {}
allFieldsForTableOfInterestColsInAllSectionsOfInterest = []
for sec in root.findall('./sections'):
columnsOfInterest = [e for e in sec.findall('columns/table/..') if isTableOfInterest(e.find('table').text, fieldsDBTableAPIName)]
if len(columnsOfInterest) > 0:
d[sec] = columnsOfInterest # Add to "d" any "section" and a list of applicable "columns" inside it
if len(d) < 1:
continue # This file is not of interest if nothing got added to "d" (if it's not a ReportType that includes any fields of the object of interest) -- move on to next file
else:
print('\r\n' + i + ', baseObject: ' + root.find('./baseObject').text) # Display which file we are working with and what its "Base Object" is
se = None
if len(d) > 0: # Why did I have this at ">1" when I found it 6 months later? Should it be >0?
allFieldsForTableOfInterestColsInAllSectionsOfInterest = [item.find('field').text for sublist in d.values() for item in sublist]
if all(fldNm in allFieldsForTableOfInterestColsInAllSectionsOfInterest for fldNm in fieldAPINames):
continue # This file is not of interest if all fields are already in it
topSections = {k for k, v in d.items() if len(v) == max([len(arr) for arr in d.values()])}
if len(topSections) >= 1:
if len(topSections) == 1:
se = (next(iter(topSections))) # There was only 1 top-ranked section -- pick it
else:
topSecsWithLabelLikeTableNewFieldIsFrom = [s for s in topSections if s.find('masterLabel').text in [fieldsDBTableAPIName, fieldsDBTableAPIName+'s']]
if len(topSecsWithLabelLikeTableNewFieldIsFrom) > 0:
se = (next(iter(topSecsWithLabelLikeTableNewFieldIsFrom))) # I don't really care which it is, honestly
else:
se = (next(iter(topSections))) # I tried my best -- moving on. Just picking a section.
else:
se = (next(iter(d.keys()))) # It was a 1-section file -- just pick the one section
if se:
uniqueTableNames = {tbstr : [e.text for e in se.findall('columns/table')].count(tbstr) for tbstr in [e.text for e in se.findall('columns/table')]}
if len(uniqueTableNames) >= 1: # We can just tack our new column onto the only section that already has other Contact values
changedAnything = False
for fieldAPIName in fieldAPINames:
if fieldAPIName not in allFieldsForTableOfInterestColsInAllSectionsOfInterest:
changedAnything = True
newColumn = ElementTree.Element('columns')
se.append(newColumn)
newColumn.append(ElementTree.Element('checkedByDefault'))
newColumn.find('checkedByDefault').text = 'false'
newColumn.append(ElementTree.Element('field'))
newColumn.find('field').text = fieldAPIName
newColumn.append(ElementTree.Element('table'))
if len(uniqueTableNames) == 1:
newColumn.find('table').text = next(iter(uniqueTableNames))
elif len(uniqueTableNames) > 1:
newColumn.find('table').text = max(uniqueTableNames, key=uniqueTableNames.get)
if changedAnything:
se[:] = sorted(se, key=lambda x: x.tag) # Put masterLabel tag back at the end of the section. We don't need to re-sort the fields because we might screw up where people were expecting to see them, if they weren't yet in alphabetical order.
root.set('xmlns','http://soap.sforce.com/2006/04/metadata')
if not os.path.exists(outpstr): os.makedirs(outpstr)
if not os.path.exists(outpstr+'reportTypes'): os.makedirs(outpstr+'reportTypes')
with open(outpstr+'reportTypes\\'+i, 'w', newline='') as fw:
dom_string = xml.dom.minidom.parseString(ElementTree.tostring(root)).toprettyxml(encoding='UTF-8', indent=' ')
dom_string = '\n'.join([s for s in dom_string.decode('UTF-8').splitlines() if s.strip()]) + '\n'
fw.write(dom_string)
changedThese.append(i[:-11])
if len(changedThese) > 0:
pkgroot = ElementTree.Element('Package', attrib={'xmlns':'http://soap.sforce.com/2006/04/metadata'})
typesElem = ElementTree.Element('types')
for x in changedThese:
membersElem = ElementTree.Element('members')
membersElem.text = x
typesElem.append(membersElem)
namesElem = ElementTree.Element('name')
namesElem.text = 'ReportType'
typesElem.append(namesElem)
pkgroot.append(typesElem)
verElem = ElementTree.Element('version')
verElem.text = metadataAPIVer
pkgroot.append(verElem)
with open(outpstr+'package.xml', 'w', newline='') as fw:
dom_string = xml.dom.minidom.parseString(ElementTree.tostring(pkgroot)).toprettyxml(encoding='UTF-8', indent=' ')
dom_string = '\n'.join([s for s in dom_string.decode('UTF-8').splitlines() if s.strip()]) + '\n'
fw.write(dom_string)
print('all done')
Wow, does one of our managed-package vendors have me riled up.
I detected a bug in their code that means an "@isTest"-flagged unit test can't INSERT a new record from various tables (including "User" records of a certain type of "Profile").
In other words, I can't write unit tests that have anything to do with several of our core tables.
I reported the bug and pointed out that:
Clearly, their triggers have some weird bugs that they never detected because they "cheated" when writing all their unit tests by adding "(SeeAllData=True)" to "@isTest" (that's the case in all the open-source work
they custom-developed for us).
Their codebase doesn't seem to be able to hold up to an empty database the way Salesforce code is supposed to, so it seems they simply make sure they're never showing it off to the customer on an empty database and have all unit tests look at "actual data" in the database.
Lovely.
So what'd they do in response to my problem?
Added "(SeeAllData=True)" to my proof-of-bug code and told me everything was all better.
*glares*
No.
It. most. definitely. is. not.
P.S. In the 29 new lines of code you "added" to my proof-of-bug unit test as well, vendor, you exposed a 2nd broken closed-source trigger when I turned off your "(SeeAllData=True)." So thanks for that, I guess -- saved me opening a 2nd ticket? *cough* #NotThatYouShouldHaveFoundItYourselfOrAnything