Normalized to: Xiao, J.
[1]
oai:arXiv.org:2006.09188 [pdf] - 2115489
A Redistribution Tool for Long-Term Archive of Astronomical Observation
Data
Submitted: 2020-06-16
Astronomical observation data require long-term preservation, and the rapid
accumulation of observation data makes it necessary to consider the cost of
long-term archive storage. In addition to low-speed disk-based online storage,
optical disk or tape-based offline storage can be used to save costs. However,
for astronomical research that requires historical data (particularly
time-domain astronomy), the performance and energy consumption of
data-accessing techniques cause problems because the requested data (which are
organized according to observation time) may be located across multiple storage
devices. In this study, we design and develop a tool referred to as AstroLayout
to redistribute the observation data using spatial aggregation. The core
algorithm uses graph partitioning to generate an optimized data placement
according to the original observation data statistics and the target storage
system. For the given observation data, AstroLayout can copy the long-term
archive in the target storage system in accordance with this placement. An
efficiency evaluation shows that AstroLayout can reduce the number of devices
activated when responding to data-access requests in time-domain astronomy
research. In addition to improving the performance of data-accessing
techniques, AstroLayout can also reduce the storage systems power consumption.
For enhanced adaptability, it supports storage systems of any media, including
optical disks, tapes, and hard disks.
[2]
oai:arXiv.org:2005.10501 [pdf] - 2099348
Towards an Astronomical Science Platform: Experiences and Lessons
Learned from Chinese Virtual Observatory
Cui, Chenzhou;
Tao, Yihan;
Li, Changhua;
Fan, Dongwei;
Xiao, Jian;
He, Boliang;
Li, Shanshan;
Yu, Ce;
Mi, Linying;
Xu, Yunfei;
Han, Jun;
Yang, Sisi;
Zhao, Yongheng;
Xue, Yanjie;
Hao, Jinxin;
Liu, Liang;
Chen, Xiao;
Chen, Junyi;
Zhang, Hailong
Submitted: 2020-05-21
In the era of big data astronomy, next generation telescopes and large sky
surveys produce data sets at the TB or even PB level. Due to their large data
volumes, these astronomical data sets are extremely difficult to transfer and
analyze using personal computers or small clusters. In order to offer better
access to data, data centers now generally provide online science platforms
that enable analysis close to the data. The Chinese Virtual Observatory
(China-VO) is one of the member projects in the International Virtual
Observatory Alliance and it is dedicated to providing a research and education
environment where globally distributed astronomy archives are simple to find,
access, and interoperate. In this study, we summarize highlights of the work
conducted at the China-VO, as well the experiences and lessons learned during
the full life-cycle management of astronomical data. Finally, We discuss the
challenges and future trends for astronomical science platforms.
[3]
oai:arXiv.org:2001.06669 [pdf] - 2112585
Deep residual detection of radio frequency interference for FAST
Submitted: 2020-01-18
Radio frequency interference (RFI) detection and excision are key steps in
the data-processing pipeline of the Five-hundred-meter Aperture Spherical radio
Telescope (FAST). Because of its high sensitivity and large data rate, FAST
requires more accurate and efficient RFI flagging methods than its
counterparts. In the last decades, approaches based upon artificial
intelligence (AI), such as codes using convolutional neural networks (CNNs),
have been proposed to identify RFI more reliably and efficiently. However, RFI
flagging of FAST data with such methods has often proved to be erroneous, with
further manual inspections required. In addition, network construction as well
as preparation of training data sets for effective RFI flagging has imposed
significant additional workloads. Therefore, rapid deployment and adjustment of
AI approaches for different observations is impractical to implement with
existing algorithms. To overcome such problems, we propose a model called
RFI-Net. With the input of raw data without any processing, RFI-Net can detect
RFI automatically, producing corresponding masks without any alteration of the
original data. Experiments with RFI-Net using simulated astronomical data show
that our model has outperformed existing methods in terms of both precision and
recall. Besides, compared with other models, our method can obtain the same
relative accuracy with fewer training data, thus reducing the effort and time
required to prepare the training data set. Further, the training process of
RFI-Net can be accelerated, with overfittings being minimized, compared with
other CNN codes. The performance of RFI-Net has also been evaluated with
observing data obtained by FAST and the Bleien Observatory. Our results
demonstrate the ability of RFI-Net to accurately identify RFI with
fine-grained, high-precision masks that required no further modification.
[4]
oai:arXiv.org:1801.07548 [pdf] - 1622532
A hybrid architecture for astronomical computing
Li, Changhua;
Cui, Chenzhou;
He, Boliang;
Fan, Dongwei;
Mi, Linying;
Li, Shanshan;
Yang, Sisi;
Xu, Yunfei;
Han, Jun;
Chen, Junyi;
Zhang, Hailong;
Yu, Ce;
Xiao, Jian;
Wang, Chuanjun;
Cao, Zihuang;
Fan, Yufeng;
Liu, Liang;
Chen, Xiao;
Song, Wenming;
Du, Kangyu
Submitted: 2018-01-18
With many large science equipment constructing and putting into use,
astronomy has stepped into the big data era. The new method and infrastructure
of big data processing has become a new requirement of many astronomers. Cloud
computing, Map/Reduce, Hadoop, Spark, etc. many new technology has sprung up in
recent years. Comparing to the high performance computing(HPC), Data is the
center of these new technology. So, a new computing architecture infrastructure
is necessary, which can be shared by both HPC and big data processing. Based on
Astronomy Cloud project of Chinese Virtual Observatory (China-VO), we have made
much efforts to optimize the designation of the hybrid computing platform.
which include the hardware architecture, cluster management, Job and Resource
scheduling.
[5]
oai:arXiv.org:1701.05641 [pdf] - 1534783
AstroCloud: A Distributed Cloud Computing and Application Platform for
Astronomy
Cui, Chenzhou;
He, Boliang;
Yu, Ce;
Xiao, Jian;
Li, Changhua;
Fan, Dongwei;
Li, Shanshan;
Mi, Linying;
Cao, Zihuang;
Yang, Sisi;
Xu, Yunfei;
Chen, Yue;
Li, Zheng;
Han, Xu;
Hong, Zhi;
Yin, Shucheng;
Li, Chen;
Wang, Chuanjun;
Fan, Yufeng;
Wang, Jianguo;
Chen, Junyi;
Zhang, Hailong;
Liu, Liang;
Gao, Na;
Yang, Zherui;
Chen, Xiao;
Liu, Min;
Qiao, Cuilan;
Du, Kangyu;
Su, Liying;
Song, Wenming
Submitted: 2017-01-19
Virtual Observatory (VO) is a data-intensively online astronomical research
and education environment, which takes advantages of advanced information
technologies to achieve seamless and global access to astronomical information.
AstroCloud is a cyber-infrastructure for astronomy research initiated by
Chinese Virtual Observatory (China-VO) project, and also a kind of physical
distributed platform which integrates lots of tasks such as telescope access
proposal management, data archiving, data quality control, data release and
open access, cloud based data processing and analysis. It consists of five
application channels, i.e. observation, data, tools, cloud and public and is
acting as a full lifecycle management system and gateway for astronomical data
and telescopes. Physically, the platform is hosted in six cities currently,
i.e. Beijing, Nanjing, Shanghai, Kunming, Lijiang and Urumqi, and serving more
than 17 thousand users. Achievements from international Virtual Observatories
and Cloud Computing are adopted heavily. In the paper, backgrounds of the
project, architecture, Cloud Computing environment, key features of the system,
current status and future plans are introduced.
[6]
oai:arXiv.org:1601.03155 [pdf] - 1340395
Update of the China-VO AstroCloud
Cui, Chenzhou;
Yu, Ce;
Xiao, Jian;
He, Boliang;
Li, Changhua;
Fan, Dongwei;
Wang, Chuanjun;
Hong, Zhi;
Li, Shanshan;
Mi, Linying;
Wan, Wanghui;
Cao, Zihuang;
Wang, Jiawei;
Yin, Shucheng;
Fan, Yufeng;
Wang, Jianguo;
Yang, Sisi;
Ling, Yin;
Zhang, Hailong;
Chen, Junyi;
Liu, Liang;
Chen, Xiao
Submitted: 2016-01-13
As the cyber-infrastructure for Astronomical research from Chinese Virtual
Observatory (China-VO) project, AstroCloud has been archived solid progresses
during the last one year. Proposal management system and data access system are
re-designed. Several new sub-systems are developed, including China-VO
PaperData, AstroCloud Statics and Public channel. More data sets and
application environments are integrated into the platform. LAMOST DR1, the
largest astronomical spectrum archive was released to the public using the
platform. The latest progresses will be introduced.
[7]
oai:arXiv.org:1507.00651 [pdf] - 1259123
Confined Flares in Solar Active Region 12192 from 2014 October 18 to 29
Submitted: 2015-07-02
Using the observations from the Atmospheric Imaging Assembly (AIA) and
Helioseismic and Magnetic Imager (HMI) aboard the Solar Dynamics Observatory
(SDO), we investigate six X-class and twenty-nine M-class flares occurring in
solar active region (AR) 12192 from October 18 to 29. Among them, thirty
(including six X- and twenty-four M-class) flares originated from the AR core
and the other five M-flares appeared at the AR periphery. Four of the X-flares
exhibited similar flaring structures, indicating they were homologous flares
with analogous triggering mechanism. The possible scenario is: photospheric
motions of emerged magnetic fluxes lead to shearing of the associated coronal
magnetic field, which then yields a tether-cutting favorable configuration.
Among the five periphery M-flares, four were associated with jet activities.
The HMI vertical magnetic field data show that the photospheric fluxes of
opposite magnetic polarities emerged, converged and canceled with each other at
the footpoints of the jets before the flares. Only one M-flare from the AR
periphery was followed by a coronal mass ejection (CME). From October 20 to 26,
the mean decay index of the horizontal background field within the height range
of 40-105 Mm is below the typical threshold for torus instability onset. This
suggests that a strong confinement from the overlying magnetic field might be
responsible for the poor CME production of AR 12192.
[8]
oai:arXiv.org:1411.5072 [pdf] - 904071
AstroCloud, a Cyber-Infrastructure for Astronomy Research: Data Access
and Interoperability
Fan, Dongwei;
He, Boliang;
Xiao, Jian;
Li, Shanshan;
Li, Changhua;
Cui, Chenzhou;
Yu, Ce;
Hong, Zhi;
Yin, Shucheng;
Wang, Chuanjun;
Cao, Zihuang;
Fan, Yufeng;
Mi, Linying;
Wan, Wanghui;
Wang, Jianguo
Submitted: 2014-11-18
Data access and interoperability module connects the observation proposals,
data, virtual machines and software. According to the unique identifier of PI
(principal investigator), an email address or an internal ID, data can be
collected by PI's proposals, or by the search interfaces, e.g. conesearch.
Files associated with the searched results could be easily transported to cloud
storages, including the storage with virtual machines, or several commercial
platforms like Dropbox. Benefitted from the standards of IVOA (International
Observatories Alliance), VOTable formatted searching result could be sent to
kinds of VO software. Latter endeavor will try to integrate more data and
connect archives and some other astronomical resources.
[9]
oai:arXiv.org:1411.5070 [pdf] - 904069
AstroCloud, a Cyber-Infrastructure for Astronomy Research: Architecture
Xiao, Jian;
Yu, Ce;
Cui, Chenzhou;
He, Boliang;
Li, Changhua;
Fan, Dongwei;
Hong, Zhi;
Yin, Shucheng;
Wang, Chuanjun;
Cao, Zihuang;
Fan, Yufeng;
Li, Shanshan;
Mi, Linying;
Wan, Wanghui;
Wang, Jianguo;
Zhang, Hailong
Submitted: 2014-11-18
AstroCloud is a cyber-Infrastructure for Astronomy Research initiated by
Chinese Virtual Observatory (China-VO) under funding support from NDRC
(National Development and Reform commission) and CAS (Chinese Academy of
Sciences). The ultimate goal of this project is to provide a comprehensive
end-to-end astronomy research environment where several independent systems
seamlessly collaborate to support the full lifecycle of the modern
observational astronomy based on big data, from proposal submission, to data
archiving, data release, and to in-situ data analysis and processing. In this
paper, the architecture and key designs of the AstroCloud platform are
introduced, including data access middleware, access control and security
framework, extendible proposal workflow, and system integration mechanism.
[10]
oai:arXiv.org:1411.5071 [pdf] - 904070
AstroCloud, a Cyber-Infrastructure for Astronomy Research: Data
Archiving and Quality Control
He, Boliang;
Cui, Chenzhou;
Fan, Dongwei;
Li, Changhua;
Xiao, Jian;
Yu, Ce;
Wang, Chuanjun;
Cao, Zihuang;
Chen, Junyi;
Yi, Weimin;
Li, Shanshan;
Mi, Linying;
Yang, Sisi
Submitted: 2014-11-18
AstroCloud is a cyber-Infrastructure for Astronomy Research initiated by
Chinese Virtual Observatory (China-VO) under funding support from NDRC
(National Development and Reform commission) and CAS (Chinese Academy of
Sciences){\url{http://astrocloud.china-vo.org}}\citep{O8-5_Cui_adassxxiv}. To
archive the astronomical data in China, we present the implementation of the
astronomical data archiving system (ADAS). Data archiving and quality control
are the infrastructure for the AstroCloud. Throughout the data of the entire
life cycle, data archiving system standardized data, transferring data, logging
observational data, archiving ambient data, And storing these data and metadata
in database. Quality control covers the whole process and all aspects of data
archiving.