Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DM-28328: Build docs in stack-os-matrix #791

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open

DM-28328: Build docs in stack-os-matrix #791

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
88aadd2
Fix typo (test pipeline).
ktlim Apr 23, 2021
9e3ddda
Update macOS versions (test pipeline).
ktlim Apr 23, 2021
b9af6d9
Add doc build/upload to stack-os-matrix.
ktlim Apr 23, 2021
3fc2344
Define platforms for doc build.
ktlim Apr 23, 2021
78d49dd
Use group command instead of subshell.
ktlim Apr 23, 2021
71f4464
Remove unneeded symlink.
ktlim Apr 28, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions etc/scipipe/build_matrix.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,9 +99,11 @@ scipipe-lsstsw-matrix:
label: osx-10.13||osx-10.14
display_name: macos
display_compiler: clang
build_docs: true
scipipe-lsstsw-lsst_distrib:
- <<: *el7-conda
- <<: *catalina-conda
build_docs: true
- <<: *bigsur-conda
scipipe-lsstsw-ci_hsc:
- <<: *el7-conda
Expand All @@ -119,6 +121,7 @@ canonical:
<<: *el7-conda
label: snowflake-0
display_name: centos-7
build_docs: true
workspace: snowflake/release
#
# eups distrib tarball configuration -- used by
Expand Down
5 changes: 2 additions & 3 deletions jobs/stack_os_matrix.groovy
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,16 +11,15 @@ p.pipeline().with {
stringParam('REFS', null, 'Whitespace delimited list of "refs" to attempt to build. Priority is highest -> lowest from left to right. "master" or branch from repos.yaml is implicitly appended to the right side of the list, if not specified; do not specify "master" explicitly.')
stringParam('PRODUCTS', scipipe.canonical.products,
'Whitespace delimited list of EUPS products to build.')
booleanParam('BUILD_DOCS', true, 'Build pipelines.lsst.io site')
booleanParam('PUBLISH_DOCS', true, 'Publish pipelines.lsst.io/v edition')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does it know where to publish that in the /v tree? Does it combine all the ticket branches that have been specified for the build? I see some code later that looks like it might be doing that and replacing space with -.

Do we clean them out periodically?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's based on the git refs, yes, and it combines all ticket branches, as requested in the ticket.

I don't think we clear them out at present. @jonathansick ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't delete old editions at present but that's a feature we have in mind for LSST the Docs.

stringParam('SPLENV_REF', scipipe.template.splenv_ref, 'conda env ref')
// XXX testing only
//booleanParam('NO_FETCH', false, 'Do not pull from git remote if branch is already the current ref. (This should generally be false outside of testing the CI system)')
}

concurrentBuild(true)

environmentVariables(
BUILD_CONFIG: 'scipipe-lsstsw-matrix',
BUILD_DOCS: false,
WIPEOUT: false,
)
}
2 changes: 1 addition & 1 deletion jobs/test_tarball.groovy
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ p.pipeline().with {
booleanParam('WIPEOUT', false, 'Completely wipe out workspace(s) before starting build.')
stringParam('TIMEOUT', '8', 'build timeout in hours')
stringParam('IMAGE', null, 'Pipeline base docker image (e.g. docker.io/lsstdm/scipipe-base:8 or empty for macOS)')
choiceParam('LABEL', ['centos-8-conda', 'centos-7-conda', 'osx-10.11', 'osx-10.12', 'osx-10.14'], 'Jenkins build agent label')
choiceParam('LABEL', ['centos-8-conda', 'centos-7-conda', 'osx-10.13', 'osx-10.14'], 'Jenkins build agent label')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is centos-8 still a thing?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For test pipelines, perhaps.

stringParam('COMPILER', 'conda-system', 'compiler version string')
choiceParam('PYTHON_VERSION', ['3'], 'Python major version')
stringParam('MINIVER', scipipe.template.tarball_defaults.miniver, 'Miniconda installer version')
Expand Down
39 changes: 39 additions & 0 deletions pipelines/lib/util.groovy
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -339,8 +339,47 @@ def lsstswBuild(
LSST_SPLENV_REF: lsstswConfig.splenv_ref
] + buildParams

if (lsstswConfig.build_docs && buildParams['LSST_BUILD_DOCS'] == "true") {
buildParams['LSST_PRODUCTS'] += " pipelines_lsst_io"
// suppress old-style doc build
buildParams['LSST_BUILD_DOCS'] = "false"
} else {
// don't publish docs if we didn't build them
buildParams['LSST_PUBLISH_DOCS'] = "false"
}

def run = {
jenkinsWrapper(buildParams)

if (buildParams['LSST_PUBLISH_DOCS'] == "true") {
withEnv(["LSST_REFS=${buildParams['LSST_REFS']}"]) {
withCredentials([[
$class: 'UsernamePasswordMultiBinding',
credentialsId: 'ltd-upload',
usernameVariable: 'LTD_USERNAME',
passwordVariable: 'LTD_PASSWORD',
]]) {
bash '''
set +o xtrace
cd lsstsw
source bin/envconfig
{
conda activate ltd &&
conda install -y -c conda-forge ltd-conveyor
} || {
conda create -y -n ltd -c conda-forge ltd-conveyor &&
conda activate ltd
}
set -o xtrace
GIT_REF=${LSST_REFS// /-}
ltd upload --product pipelines --dir build/pipelines_lsst_io/_build/html --git-ref "$GIT_REF"
set +o xtrace
conda deactivate
set -o xtrace
'''
} // withCredentials
} // withEnv
} // if LSST_PUBLISH_DOCS
} // run

def runDocker = {
Expand Down
2 changes: 2 additions & 0 deletions pipelines/stack_os_matrix.groovy
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,13 +23,15 @@ notify.wrap {
'BUILD_CONFIG',
'PRODUCTS',
'BUILD_DOCS',
'PUBLISH_DOCS',
'WIPEOUT',
])

def buildParams = [
LSST_REFS: REFS,
LSST_PRODUCTS: PRODUCTS,
LSST_BUILD_DOCS: BUILD_DOCS,
LSST_PUBLISH_DOCS: PUBLISH_DOCS,
]

// override conda env ref from build_matrix.yaml
Expand Down
2 changes: 1 addition & 1 deletion pipelines/test_stack_os_matrix.groovy
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ notify.wrap {
if (params.RUBINENV_ORG_FORK != null) {
buildParams['RUBINENV_ORG_FORK'] = params.RUBINENV_ORG_FORK
}
if (params.RUBINENV_BRANCHi != null) {
if (params.RUBINENV_BRANCH != null) {
buildParams['RUBINENV_BRANCH'] = params.RUBINENV_BRANCH
}

Expand Down